ove / ove-asset-services Goto Github PK

View Code? Open in Web Editor NEW

1.0 6.0 1.0 5.65 MB

Asset Management for OVE

License: MIT License

Dockerfile 2.85% C# 74.69% CSS 1.63% JavaScript 0.08% Shell 2.56% HTML 18.19%

docker ove dotnet-core csharp ove-services

ove-asset-services's Introduction

⛔️ DEPRECATED This repository has been archived in favour of the new AM Service

Open Visualisation Environment - Asset Services

This collection of services provides end users with a simple way of managing and processing content to be displayed using Open Visualisation Environment (OVE).

OVE is an open-source software stack, designed to be used in large high resolution display (LHRD) environments like the Imperial College Data Science Institute's Data Observatory.

We welcome collaboration under our Code of Conduct.

The OVE Documentation provides installation instructions and a user guide.

This repository contains an Asset Manager and a collection of Asset Processing Services:

Image Tiles - creates Deep Zoom image tilesets from Image based Assets.

ove-asset-services's People

Contributors

Stargazers

Watchers

Forkers

senakafdo

ove-asset-services's Issues

Rename Repo

I suggest this repo is called ove-asset-services since it contains more than just an asset manager but services which support working with assets.

Setup & test the routing to latest dotnet version

services.AddMvc(options => {
    options.InputFormatters.Insert(0, new RawRequestBodyFormatter());
})
.SetCompatibilityVersion(CompatibilityVersion.Latest)
.AddXmlSerializerFormatters().AddJsonOptions(options => {
   options.SerializerSettings.Formatting = Formatting.Indented;
}).AddRazorPagesOptions( o=> {
         ...
});
}

Change filename validation to url encoding

as requested by @DavidAkroyd

Unit tests for all the sevices where this makes sense

An example of this is code used validate S3 paths, bucket names etc.

Provide a new Graph Tile Service

Docker build instructions are inaccurate

Currently we have Alternatively, you can use docker. Each package has its own docker image, which can be build by executing the build script or docker-compose directly.

However, we don't use docker-compose at all within this project, and we also do not provide instructions on how to run the build scripts. It is probably worth to note that linux tools need to be installed in windows environments in order to run these build scripts.

C# 1591 warnings during the build

The build generates a number of .NET warnings around missing documentation:

Controllers/HomeController.cs(8,18): warning CS1591: Missing XML comment for publicly visible type or member 'HomeController' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Controllers/HomeController.cs(12,16): warning CS1591: Missing XML comment for publicly visible type or member 'HomeController.HomeController(ILogger)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Controllers/HomeController.cs(15,30): warning CS1591: Missing XML comment for publicly visible type or member 'HomeController.Index()' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Controllers/HomeController.cs(22,30): warning CS1591: Missing XML comment for publicly visible type or member 'HomeController.Error()' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Controllers/ServicesRegistryController.cs(16,16): warning CS1591: Missing XML comment for publicly visible type or member 'ServicesRegistryController.ServicesRegistryController(ILogger, ServiceRepository)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
DbContexts/AssetModelContext.cs(8,18): warning CS1591: Missing XML comment for publicly visible type or member 'AssetModelContext' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
DbContexts/AssetModelContext.cs(13,16): warning CS1591: Missing XML comment for publicly visible type or member 'AssetModelContext.AssetModelContext(DbContextOptions)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
DbContexts/AssetModelContext.cs(16,37): warning CS1591: Missing XML comment for publicly visible type or member 'AssetModelContext.AssetModels' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
DbContexts/AssetModelContext.cs(18,28): warning CS1591: Missing XML comment for publicly visible type or member 'AssetModelContext.Initialize(IServiceProvider)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/ControllerExtensions.cs(4,25): warning CS1591: Missing XML comment for publicly visible type or member 'ControllerExtensions' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/IAssetFileOperations.cs(10,16): warning CS1591: Missing XML comment for publicly visible type or member 'IAssetFileOperations.ResolveFileUrl(OVEAssetModel)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/IAssetFileOperations.cs(12,20): warning CS1591: Missing XML comment for publicly visible type or member 'IAssetFileOperations.Move(OVEAssetModel, OVEAssetModel)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/IAssetFileOperations.cs(13,20): warning CS1591: Missing XML comment for publicly visible type or member 'IAssetFileOperations.Delete(OVEAssetModel)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/IAssetFileOperations.cs(14,20): warning CS1591: Missing XML comment for publicly visible type or member 'IAssetFileOperations.Save(OVEAssetModel, IFormFile)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/S3AssetFileOperations.cs(17,18): warning CS1591: Missing XML comment for publicly visible type or member 'S3AssetFileOperations' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/S3AssetFileOperations.cs(26,16): warning CS1591: Missing XML comment for publicly visible type or member 'S3AssetFileOperations.S3AssetFileOperations(ILogger, IConfiguration)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/S3AssetFileOperations.cs(48,23): warning CS1591: Missing XML comment for publicly visible type or member 'S3AssetFileOperations.ResolveFileUrl(OVEAssetModel)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/S3AssetFileOperations.cs(55,33): warning CS1591: Missing XML comment for publicly visible type or member 'S3AssetFileOperations.Move(OVEAssetModel, OVEAssetModel)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/S3AssetFileOperations.cs(62,33): warning CS1591: Missing XML comment for publicly visible type or member 'S3AssetFileOperations.Delete(OVEAssetModel)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/S3AssetFileOperations.cs(100,33): warning CS1591: Missing XML comment for publicly visible type or member 'S3AssetFileOperations.Save(OVEAssetModel, IFormFile)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/ServiceRepository.cs(21,16): warning CS1591: Missing XML comment for publicly visible type or member 'ServiceRepository.ServiceRepository(ILogger, IConfiguration)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/ServiceRepository.cs(34,37): warning CS1591: Missing XML comment for publicly visible type or member 'ServiceRepository.GetServices()' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/ServiceRepository.cs(38,21): warning CS1591: Missing XML comment for publicly visible type or member 'ServiceRepository.ValidateServiceChoice(string, IFormFile)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/ServiceRepository.cs(43,21): warning CS1591: Missing XML comment for publicly visible type or member 'ServiceRepository.ValidateServiceChoice(string, string)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/ServiceRepository.cs(47,21): warning CS1591: Missing XML comment for publicly visible type or member 'ServiceRepository.UpdateService(OVEService)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/ServiceRepository.cs(52,27): warning CS1591: Missing XML comment for publicly visible type or member 'ServiceRepository.GetService(string)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Domain/ServiceRepository.cs(79,23): warning CS1591: Missing XML comment for publicly visible type or member 'ServiceRepository.GetViewUrl(OVEAssetModel)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Migrations/20181009081536_InitialCreate.cs(7,26): warning CS1591: Missing XML comment for publicly visible type or member 'InitialCreate' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Migrations/20181009081536_InitialCreate.cs(9,33): warning CS1591: Missing XML comment for publicly visible type or member 'InitialCreate.Up(MigrationBuilder)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Migrations/20181009081536_InitialCreate.cs(32,33): warning CS1591: Missing XML comment for publicly visible type or member 'InitialCreate.Down(MigrationBuilder)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Migrations/20181009081536_InitialCreate.Designer.cs(15,33): warning CS1591: Missing XML comment for publicly visible type or member 'InitialCreate.BuildTargetModel(ModelBuilder)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Models/ErrorViewModel.cs(3,18): warning CS1591: Missing XML comment for publicly visible type or member 'ErrorViewModel' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Models/ErrorViewModel.cs(4,23): warning CS1591: Missing XML comment for publicly visible type or member 'ErrorViewModel.RequestId' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Models/ErrorViewModel.cs(6,21): warning CS1591: Missing XML comment for publicly visible type or member 'ErrorViewModel.ShowRequestId' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Models/OVEAssetModel.cs(17,23): warning CS1591: Missing XML comment for publicly visible type or member 'OVEAssetModel.Id' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Models/OVEAssetModel.cs(23,23): warning CS1591: Missing XML comment for publicly visible type or member 'OVEAssetModel.Project' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Models/OVEAssetModel.cs(28,23): warning CS1591: Missing XML comment for publicly visible type or member 'OVEAssetModel.Name' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Models/OVEAssetModel.cs(30,23): warning CS1591: Missing XML comment for publicly visible type or member 'OVEAssetModel.Description' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Models/OVEAssetModel.cs(33,25): warning CS1591: Missing XML comment for publicly visible type or member 'OVEAssetModel.LastModified' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Models/OVEAssetModel.cs(64,20): warning CS1591: Missing XML comment for publicly visible type or member 'OVEAssetModel.ProcessingState' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Models/OVEAssetModel.cs(72,32): warning CS1591: Missing XML comment for publicly visible type or member 'OVEAssetModel.ToString()' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Program.cs(13,18): warning CS1591: Missing XML comment for publicly visible type or member 'Program' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Program.cs(14,28): warning CS1591: Missing XML comment for publicly visible type or member 'Program.Main(string[])' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Startup.cs(19,18): warning CS1591: Missing XML comment for publicly visible type or member 'Startup' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Startup.cs(20,16): warning CS1591: Missing XML comment for publicly visible type or member 'Startup.Startup(IConfiguration)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Startup.cs(37,21): warning CS1591: Missing XML comment for publicly visible type or member 'Startup.ConfigureServices(IServiceCollection)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]
Startup.cs(94,21): warning CS1591: Missing XML comment for publicly visible type or member 'Startup.Configure(IApplicationBuilder, IHostingEnvironment)' [/usr/src/app/src/OVE.Service.AssetManager/OVE.Service.AssetManager.csproj]

Test update to NetVips to v1.0.5 on Docker Image

@oserban, can you please test and update the dependency version accordingly. I wanted to understand whether the NetVips 1.0.5 issue was something local to @davidbirchwork or not. If it is local to @davidbirchwork, and non reproducible within the container, please apply the PR suggested by Renovate in #1.

Asset manager raises a critical error reading the home controller

ovehub-ove-asset-manager_1       | crit: OVE.Service.AssetManager.Controllers.HomeController[0]
ovehub-ove-asset-manager_1       |       Read Index page

Extract common text from the Configuration sections of README.md files

Extract common sections such as Configuring S3 and common text such as the modifications to appsettings.json into a separate file following the DRY principal.

Provide SQL database and port asset manger to use SQL service

/api/ServicesRegistry/GetService returns details of non-existing Graph Service

If someone only starts the Asset Manager with the ImageTiles service, they should not see details about the non-existing Graph service. The services should ideally be registering themselves with the Asset Manager, according to the design of it.

Make Swagger URLs consistent

As discussed, lets also use HOST:PORT/api-docs as the Swagger endpoint across the board.

Integration tests for the asset manager

If the tests are successful, travis needs to retag the components from latest-unstable to latest

Setup MariaDB to work with an external volume

Currently it saves the data within the docker image

The image tile service should not die if the registration procedure fails

If the tile service started faster than the asset manager, the tile service dies and gets restarted by pm2.

I think a better approach is to sleep for x ms and then retry, making y attempts to register. It makes the whole ecosystem more stable and less error prone on bootup.

Investigate why the asset manager is not starting on port 8080 in docker

Graph layout service

Sigma.js provides extremely limited abilities for graph layout: it has a plugin that implements the forceAtlas2 algorithm, and a plugin that arranges the nodes in a grid then repulses any nodes that overlap, but nothing else.

This means that a user who wants an alternative layout must first perform the layout initially before using the OVE Networks app.

We could address this by adding a OVE Aset Service that accepts a network and the name of layout algorithm (with values for any required parameters), and computes the layout. This could use either:

Gephi (probably using the Gephi Toolkit, which provides a library as a single JAR file)
Tulip (probably using the python bindings, though it can also be used as a C++ library)

Images not processed when running OVE Asset Services with Docker

Steps to reproduce:

Edit the docker-compose.yml inside each individual asset service and set the environment variables such as S3 Client Access keys, secret etc.
Build each individual asset service running the ./build.sh
Run each individual asset service using export SERVICE_VERSION=latest; docker-compose -f docker-compose.yml up -d
Navigate to http://localhost:8181/OVEAssetModelController/Create and upload a new image. It remains indefinitely in the Unprocessed state.

This issue is blocking the successful testing of many pull requests that have been submitted and therefore delays their approval.

Potentially unwanted Thread.Sleep(3000) in code

As discussed with @davidbirchwork this seems to be a wait to avoid a directory not being created on time for use by external services. But the operation used is documented as a synchronous operation and therefore, a wait should not be required. However, there may be something else that leads to this which can be dealt with in some other way.

[Image Tiles Service] Consistency bug: AssetManagerHost vs ServiceHostUrl in appsettings.json

We should have either AssetManagerHost and ServiceHost or AssetManagerHostUrl and ServiceHostUrl.

Add support for .zip archives

as requested by @DavidAkroyd & @senakafdo

Inject Logging into Startup Dependency Injection before it is properly Configured

Discussed already in PR #37.

Unable to upload images to Object Store

Serious issue WRT uploading images to Object Store - means nothing new can now added now.

fail: OVE.Service.AssetManager.Controllers.HomeController[0]
Viewing Error: OVE.Service.AssetManager.Models.ErrorViewModel Request: Microsoft.AspNetCore.Http.Internal.DefaultHttpRequest
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
An unhandled exception has occurred while executing the request.
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Request body too large.
at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.ForContentLength.OnReadStarting()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody.TryInit()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody.ReadAsync(Memory1 buffer, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory1 buffer, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.BufferedReadStream.EnsureBufferedAsync(Int32 minCount, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool1 bytePool, Nullable1 limit, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.MultipartReader.ReadNextSectionAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Mvc.ModelBinding.FormValueProviderFactory.AddValueProviderAsync(ValueProviderFactoryContext context)
at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.CreateAsync(ActionContext actionContext, IList`1 factories)
at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.CreateAsync(ControllerContext controllerContext)
at Microsoft.AspNetCore.Mvc.Internal.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<g__Bind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
fail: OVE.Service.AssetManager.Controllers.HomeController[0]

Provide documentation for the Asset Services on a per-service basis

There are couple of challenges right now, around where to set the credentials and how to run the new Docker build, which was the basis of creating this issue. But, we also need to explain:

What each service does
How to configure each service (S3 details, hostnames etc).
How to setup and configure the dependencies (MariaDB, ObjectStore etc) and instructions on how to use alternative DBs (SQL Server, SQLite, Postgres, etc) or object stores.
How to find the configuration parameters for each service (i.e. link to S3 docs on where in S3 you can find access key, secret etc)
How the UI works
How to use the APIs to obtain locations of where assets were uploaded (along with sample curl commands for Linux/Mac and Windows environments)