DSpace Replication Task Suite - Backup/restore DSpace to/from DuraCloud or mounted drive
Home Page: https://wiki.lyrasis.org/display/DSPACE/ReplicationTaskSuite
License: BSD 3-Clause "New" or "Revised" License
DSpace Replication Task Suite ============================= The Replication Task Suite is an add-on for DSpace which allows you to more easily replicate / backup DSpace content to another location. It utilizes the DSpace Curation System to allow all backup/restore/auditing tasks to be performed either via Command Line (CLI) or from the DSpace Admin User Interface. Replication / Backup Locations ------------------------------- Supported Replication / Backup locations include: * Local filesystem folder * Mounted drive * DuraCloud (http://www.duracloud.org) Backup Format ------------- DSpace content is replicated / backed up as a set of AIPs (Archival Information Packages). The Replication Task Suite supports two different AIP formats: 1. The DSpace AIP Format based on METS (Default) https://wiki.lyrasis.org/display/DSDOC6x/DSpace+AIP+Format 2. A BagIt based AIP Format (Beta) Sample Supported Actions ------------------------ The Replication Task Suite allows you to backup DSpace content, restore/replace DSpace content (from a backup), or audit your existing backup. Some sample actions included: * Replicate / Back up either on demand (via Admin UI or CLI) or on a scheduled basis (using a cron job) * Restore / Replace content in DSpace based on current backups (via Admin UI or CLI) * Perform checksum audit of your replicated content (via Admin UI or CLI) * Provide a rough estimate of the amount of storage space your AIPs will require (via Admin UI or CLI) * Keep a running "odometer" of all Input / Output transfers (via Admin UI or CLI) More Information ---------------- Installation documentation, configuration documentation and usage examples are all available at: https://wiki.lyrasis.org/display/DSPACE/ReplicationTaskSuite
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
The process transfers the Collection and Community zip files to a space in Duracloud before it stops and the following error appears in the logs. We're using version 3.4 of dspace-replicate and DSpace 5.8.
2017-11-09 13:41:34,590 ERROR org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter @ Serious Error Occurred Processing Request!
org.spriagframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.NoSuchMethodError: org.dspace.content.Item.getMetadata(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)[Lorg/dspace/content/DCValue;
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:972)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.dspace.app.xmlui.cocoon.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter.doFilter(DSpaceCocoonServletFilter.java:274)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.dspace.app.xmlui.cocoon.servlet.multipart.DSpaceMultipartFilter.doFilter(DSpaceMultipartFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.dspace.utils.servlet.DSpaceWebappServletFilter.doFilter(DSpaceWebappServletFilter.java:78)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: org.dspace.content.Item.getMetadata(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)[Lorg/dspace/content/DCValue;
at org.dspace.pack.bagit.ItemPacker.pack(ItemPacker.java:96)
at org.dspace.ctask.replicate.TransmitAIP.perform(TransmitAIP.java:65)
at org.dspace.curate.ResolvedTask.perform(ResolvedTask.java:88)
at org.dspace.curate.Curator$TaskRunner.run(Curator.java:563)
The change (DSpace/DSpace@1c911e2) only applies to version 6.x of DSpace, so it only affects the master branch of dspace-replicate.
Hi, we have noticed a difference in behaviour between LocalObjectStore and MountableObjectStore in the transferObject method. The local storage plugin moves the files from the replica store in the aip store as expected, whereas the mountable storage plugin leaves a redundant copy in the replica store.
While releasing RTS version 7.0, I discovered that it won't compile if I try to use it with DSpace 7.1 (or later). Therefore RTS 7.0 is only compatible with DSpace 7.0.
Here's the compilation issues: https://github.com/DSpace/dspace-replicate/runs/4241502698?check_suite_focus=true
Here's the change I made to get it to support DSpace 7.1 or later: 6c1dd04 (the change is the one to the dspace.version tag)
Maybe I am missing it, but I do not believe there is a way to set the store-id flag for the duracloud sync portion.
I believe this is needed to be able to upload directly to Chronopolis which is hosted on the same platform as Duracloud.
From what I can tell is if RTS is enabled and unable to connect to duracloud it breaks authentication such as Shibboleth
Not sure if there is a way to gracefully fail instead of throwing exceptions when duracloud isn't available.
This is at least on DSpace 7.6.2
dspace | 2024-08-22 23:11:50,315 ERROR unknown unknown org.dspace.authenticate.ShibAuthentication @ Unable to successfully authenticate using shibboleth for user because of an exception.
dspace | java.lang.IllegalStateException: Unable to aquire dispatcher named default
dspace | at org.dspace.event.EventServiceImpl.getDispatcher(EventServiceImpl.java:110) ~[dspace-api-7.6.2.jar:7.6.2]
dspace | at org.dspace.core.Context.dispatchEvents(Context.java:470) ~[dspace-api-7.6.2.jar:7.6.2]
dspace | at org.dspace.authenticate.ShibAuthentication.updateEPerson(ShibAuthentication.java:879) ~[dspace-api-7.6.2.jar:7.6.2]
dspace | at org.dspace.authenticate.ShibAuthentication.authenticate(ShibAuthentication.java:233) [dspace-api-7.6.2.jar:7.6.2]
dspace | at org.dspace.authenticate.AuthenticationServiceImpl.authenticateInternal(AuthenticationServiceImpl.java:108) [dspace-api-7.6.2.jar:7.6.2]
dspace | at org.dspace.authenticate.AuthenticationServiceImpl.authenticate(AuthenticationServiceImpl.java:80) [dspace-api-7.6.2.jar:7.6.2]
dspace | at org.dspace.app.rest.security.EPersonRestAuthenticationProvider.authenticateNewLogin(EPersonRestAuthenticationProvider.java:130) [classes/:7.6.2]
dspace | at org.dspace.app.rest.security.EPersonRestAuthenticationProvider.authenticate(EPersonRestAuthenticationProvider.java:88) [classes/:7.6.2]
dspace | at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:182) [spring-security-core-5.7.11.jar:5.7.11]
dspace | at org.dspace.app.rest.security.ShibbolethLoginFilter.attemptAuthentication(ShibbolethLoginFilter.java:78) [classes/:7.6.2]
dspace | at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.dspace.app.rest.security.StatelessAuthenticationFilter.doFilterInternal(StatelessAuthenticationFilter.java:102) [classes/:7.6.2]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:117) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:112) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:82) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:221) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) [catalina.jar:9.0.93]
dspace | at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) [catalina.jar:9.0.93]
dspace | at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) [spring-boot-actuator-2.7.18.jar:2.7.18]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) [catalina.jar:9.0.93]
dspace | at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:126) [spring-boot-2.7.18.jar:2.7.18]
dspace | at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:64) [spring-boot-2.7.18.jar:2.7.18]
dspace | at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:101) [spring-boot-2.7.18.jar:2.7.18]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:119) [spring-boot-2.7.18.jar:2.7.18]
dspace | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) [catalina.jar:9.0.93]
dspace | at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) [catalina.jar:9.0.93]
dspace | at org.springframework.web.filter.ForwardedHeaderFilter.doFilterInternal(ForwardedHeaderFilter.java:156) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) [catalina.jar:9.0.93]
dspace | at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:70) [log4j-web-2.23.1.jar:2.23.1]
dspace | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:660) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) [catalina.jar:9.0.93]
dspace | at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:383) [tomcat-coyote.jar:9.0.93]
dspace | at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) [tomcat-coyote.jar:9.0.93]
dspace | at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:936) [tomcat-coyote.jar:9.0.93]
dspace | at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791) [tomcat-coyote.jar:9.0.93]
dspace | at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) [tomcat-coyote.jar:9.0.93]
dspace | at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) [tomcat-util.jar:9.0.93]
dspace | at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-util.jar:9.0.93]
dspace | at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) [tomcat-util.jar:9.0.93]
dspace | at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
dspace | Caused by: java.io.IOException: Unable to connect to the DuraCloud Primary Content Store. Please check the DuraCloud connection/authentication settings in your 'duracloud.cfg' file.
dspace | at org.dspace.ctask.replicate.store.DuraCloudObjectStore.init(DuraCloudObjectStore.java:83) ~[dspace-replicate-7.6.jar:7.6]
dspace | at org.dspace.ctask.replicate.ReplicaManager.<init>(ReplicaManager.java:75) ~[dspace-replicate-7.6.jar:7.6]
dspace | at org.dspace.ctask.replicate.ReplicaManager.instance(ReplicaManager.java:95) ~[dspace-replicate-7.6.jar:7.6]
dspace | at org.dspace.ctask.replicate.METSReplicateConsumer.initialize(METSReplicateConsumer.java:127) ~[dspace-replicate-7.6.jar:7.6]
dspace | at org.dspace.event.EventServiceImpl$DispatcherPoolFactory.makeObject(EventServiceImpl.java:198) ~[dspace-api-7.6.2.jar:7.6.2]
dspace | at org.dspace.event.EventServiceImpl$DispatcherPoolFactory.makeObject(EventServiceImpl.java:151) ~[dspace-api-7.6.2.jar:7.6.2]
dspace | at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:796) ~[commons-pool2-2.12.0.jar:2.12.0]
dspace | at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:444) ~[commons-pool2-2.12.0.jar:2.12.0]
dspace | at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:355) ~[commons-pool2-2.12.0.jar:2.12.0]
dspace | at org.apache.commons.pool2.PoolUtils$SynchronizedKeyedObjectPool.borrowObject(PoolUtils.java:732) ~[commons-pool2-2.12.0.jar:2.12.0]
dspace | at org.dspace.event.EventServiceImpl.getDispatcher(EventServiceImpl.java:108) ~[dspace-api-7.6.2.jar:7.6.2]
dspace | ... 86 more
dspace | Caused by: org.duracloud.error.ContentStoreException: Error retrieving content stores. Response code was 401, expected value was 200. Response Body: <!doctype html><html lang="en"><head><title>HTTP Status 401 – Unauthorized</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 401 – Unauthorized</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> Unauthorized</p><p><b>Description</b> The request has not been applied to the target resource because it lacks valid authentication credentials for that resource.</p><hr class="line" /><h3>Apache Tomcat/9.0.82</h3></body></html>Error retrieving content stores. Response code was 401, expected value was 200. Response Body: <!doctype html><html lang="en"><head><title>HTTP Status 401 – Unauthorized</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 401 – Unauthorized</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> Unauthorized</p><p><b>Description</b> The request has not been applied to the target resource because it lacks valid authentication credentials for that resource.</p><hr class="line" /><h3>Apache Tomcat/9.0.82</h3></body></html>
dspace | at org.duracloud.client.ContentStoreManagerImpl.getStorageAccounts(ContentStoreManagerImpl.java:219) ~[storeclient-7.0.0.jar:?]
dspace | at org.duracloud.client.ContentStoreManagerImpl.getPrimaryContentStore(ContentStoreManagerImpl.java:157) ~[storeclient-7.0.0.jar:?]
dspace | at org.duracloud.client.ContentStoreManagerImpl.getPrimaryContentStore(ContentStoreManagerImpl.java:149) ~[storeclient-7.0.0.jar:?]
dspace | at org.dspace.ctask.replicate.store.DuraCloudObjectStore.init(DuraCloudObjectStore.java:81) ~[dspace-replicate-7.6.jar:7.6]
dspace | at org.dspace.ctask.replicate.ReplicaManager.<init>(ReplicaManager.java:75) ~[dspace-replicate-7.6.jar:7.6]
dspace | at org.dspace.ctask.replicate.ReplicaManager.instance(ReplicaManager.java:95) ~[dspace-replicate-7.6.jar:7.6]
dspace | at org.dspace.ctask.replicate.METSReplicateConsumer.initialize(METSReplicateConsumer.java:127) ~[dspace-replicate-7.6.jar:7.6]
dspace | at org.dspace.event.EventServiceImpl$DispatcherPoolFactory.makeObject(EventServiceImpl.java:198) ~[dspace-api-7.6.2.jar:7.6.2]
dspace | at org.dspace.event.EventServiceImpl$DispatcherPoolFactory.makeObject(EventServiceImpl.java:151) ~[dspace-api-7.6.2.jar:7.6.2]
dspace | at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:796) ~[commons-pool2-2.12.0.jar:2.12.0]
dspace | at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:444) ~[commons-pool2-2.12.0.jar:2.12.0]
dspace | at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:355) ~[commons-pool2-2.12.0.jar:2.12.0]
dspace | at org.apache.commons.pool2.PoolUtils$SynchronizedKeyedObjectPool.borrowObject(PoolUtils.java:732) ~[commons-pool2-2.12.0.jar:2.12.0]
dspace | at org.dspace.event.EventServiceImpl.getDispatcher(EventServiceImpl.java:108) ~[dspace-api-7.6.2.jar:7.6.2]
dspace | ... 86 more
dspace | Caused by: org.duracloud.storage.error.StorageException: Error retrieving content stores. Response code was 401, expected value was 200. Response Body: <!doctype html><html lang="en"><head><title>HTTP Status 401 – Unauthorized</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 401 – Unauthorized</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> Unauthorized</p><p><b>Description</b> The request has not been applied to the target resource because it lacks valid authentication credentials for that resource.</p><hr class="line" /><h3>Apache Tomcat/9.0.82</h3></body></html>
dspace | at org.duracloud.client.ContentStoreManagerImpl.getStorageAccounts(ContentStoreManagerImpl.java:216) ~[storeclient-7.0.0.jar:?]
dspace | at org.duracloud.client.ContentStoreManagerImpl.getPrimaryContentStore(ContentStoreManagerImpl.java:157) ~[storeclient-7.0.0.jar:?]
dspace | at org.duracloud.client.ContentStoreManagerImpl.getPrimaryContentStore(ContentStoreManagerImpl.java:149) ~[storeclient-7.0.0.jar:?]
dspace | at org.dspace.ctask.replicate.store.DuraCloudObjectStore.init(DuraCloudObjectStore.java:81) ~[dspace-replicate-7.6.jar:7.6]
dspace | at org.dspace.ctask.replicate.ReplicaManager.<init>(ReplicaManager.java:75) ~[dspace-replicate-7.6.jar:7.6]
dspace | at org.dspace.ctask.replicate.ReplicaManager.instance(ReplicaManager.java:95) ~[dspace-replicate-7.6.jar:7.6]
dspace | at org.dspace.ctask.replicate.METSReplicateConsumer.initialize(METSReplicateConsumer.java:127) ~[dspace-replicate-7.6.jar:7.6]
dspace | at org.dspace.event.EventServiceImpl$DispatcherPoolFactory.makeObject(EventServiceImpl.java:198) ~[dspace-api-7.6.2.jar:7.6.2]
dspace | at org.dspace.event.EventServiceImpl$DispatcherPoolFactory.makeObject(EventServiceImpl.java:151) ~[dspace-api-7.6.2.jar:7.6.2]
dspace | at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:796) ~[commons-pool2-2.12.0.jar:2.12.0]
dspace | at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:444) ~[commons-pool2-2.12.0.jar:2.12.0]
dspace | at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:355) ~[commons-pool2-2.12.0.jar:2.12.0]
dspace | at org.apache.commons.pool2.PoolUtils$SynchronizedKeyedObjectPool.borrowObject(PoolUtils.java:732) ~[commons-pool2-2.12.0.jar:2.12.0]
dspace | at org.dspace.event.EventServiceImpl.getDispatcher(EventServiceImpl.java:108) ~[dspace-api-7.6.2.jar:7.6.2]
dspace | ... 86 more
dspace | 2024-08-22 23:11:50,330 ERROR unknown unknown org.dspace.app.rest.security.StatelessLoginFilter @ Authentication failed (status:401)
dspace | org.springframework.security.authentication.BadCredentialsException: Login failed
dspace | at org.dspace.app.rest.security.EPersonRestAuthenticationProvider.authenticateNewLogin(EPersonRestAuthenticationProvider.java:150) ~[classes/:7.6.2]
dspace | at org.dspace.app.rest.security.EPersonRestAuthenticationProvider.authenticate(EPersonRestAuthenticationProvider.java:88) ~[classes/:7.6.2]
dspace | at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:182) ~[spring-security-core-5.7.11.jar:5.7.11]
dspace | at org.dspace.app.rest.security.ShibbolethLoginFilter.attemptAuthentication(ShibbolethLoginFilter.java:78) ~[classes/:7.6.2]
dspace | at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.dspace.app.rest.security.StatelessAuthenticationFilter.doFilterInternal(StatelessAuthenticationFilter.java:102) [classes/:7.6.2]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:117) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:112) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:82) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:221) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186) [spring-security-web-5.7.11.jar:5.7.11]
dspace | at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) [catalina.jar:9.0.93]
dspace | at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) [catalina.jar:9.0.93]
dspace | at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) [spring-boot-actuator-2.7.18.jar:2.7.18]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) [catalina.jar:9.0.93]
dspace | at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:126) [spring-boot-2.7.18.jar:2.7.18]
dspace | at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:64) [spring-boot-2.7.18.jar:2.7.18]
dspace | at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:101) [spring-boot-2.7.18.jar:2.7.18]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:119) [spring-boot-2.7.18.jar:2.7.18]
dspace | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) [catalina.jar:9.0.93]
dspace | at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) [catalina.jar:9.0.93]
dspace | at org.springframework.web.filter.ForwardedHeaderFilter.doFilterInternal(ForwardedHeaderFilter.java:156) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.34.jar:5.3.34]
dspace | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) [catalina.jar:9.0.93]
dspace | at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:70) [log4j-web-2.23.1.jar:2.23.1]
dspace | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:660) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [catalina.jar:9.0.93]
dspace | at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) [catalina.jar:9.0.93]
dspace | at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:383) [tomcat-coyote.jar:9.0.93]
dspace | at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) [tomcat-coyote.jar:9.0.93]
dspace | at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:936) [tomcat-coyote.jar:9.0.93]
dspace | at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791) [tomcat-coyote.jar:9.0.93]
dspace | at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) [tomcat-coyote.jar:9.0.93]
dspace | at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) [tomcat-util.jar:9.0.93]
dspace | at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-util.jar:9.0.93]
dspace | at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) [tomcat-util.jar:9.0.93]
dspace | at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
Imported from JIRA [DS-4556] created by bbranan
This task was split out of DS-2000
We should enhance the error handling in the Replication Task Suite so that it's possible to report individual backup failures, but continue the backup process. We should not always return a complete failure if a single error is encountered...instead we should backup what content we can and report which content failed to be backed up.
Imported from JIRA [DS-4312] created by mspalti
Transmitting AIPs to DuraCloud that are larger than 5GB results in a 504 error due to limitations imposed by Amazon. This causes the backup to halt. There's a discussion of changing this behavior (see DS-4309) by allowing curation tasks to continue and reporting any failures. Even with that, I think there's a case here for adding a skip list for ignoring AIPs that are known to be too large (it would be even better to introduce chunking so that RTS could transmit these larger objects!).
Imported from JIRA [DS-4458] created by [email protected]
While adding the bagit-support library in the Replication Task Suite, support for holey bags by using the fetch.txt was removed. This is because the bagit-support library does not support creating bags with a fetch.txt.
We should see how much interest there is in either keeping/re-adding support for this feature, and implement it depending on the responses.
We found that exporting an ITEM is reliably generating otherIds with an additional missing character.
bagType AIP
objectType item
objectId 1957/59277
ownerId 1957/4
otherIds 1957/4390
otherIds should be 1957/43909 in this case.
I'm supposing that the following line is taking one too many characters:
fwriter.writeProperty(OTHER_IDS, linkedStr.substring(0, linkedStr.length() - 2));
should probably be
fwriter.writeProperty(OTHER_IDS, linkedStr.substring(0, linkedStr.length() - 1));
I have been testing locally with both export formats: METS and Bagit and when using the Bagit format the Audit curation task always fails.
Steps to reproduce:
The curation results always report differences in checkums.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.