Giter Club home page Giter Club logo

topology's Introduction

Topology Build Status

This repository contains the registry of OSG projects, site resources, and Virtual Organizations, as well as scripts for managing and displaying this information.

For registration instructions, please see this document.

The information was sourced from the former OIM site that was hosted at oim.grid.iu.edu. This README contains the following sections:

Structure of the Registry

The data is organized as files in YAML format as follows:

  • projects/ contains information about research projects that run jobs on OSG. Each project has its own file, named <PROJECT>.yaml

  • virtual-organizations/ contains information about Virtual Organizations (VOs). Each VO has its own file, named <VO>.yaml. Additionally, there is a REPORTING_GROUPS.yaml file that contains information about WLCG metric reporting groups for each VO.

  • topology/ contains information about the topology of the resources that OSG sites provide. Resources such as HTCondor-CE, GridFTP, XRootD, or Squid are collected into "resource groups." Each resource group has its own file, which are further organized by facility, and then by site. Resource group files are named <FACILITY>/<SITE>/<RESOURCEGROUP>.yaml

    For example, the OSG resources in the Center for High Throughput Computing can be found in the following file: topology/University of Wisconsin/CHTC/CHTC.yaml

Viewing the Registry

Registry data can be accessed in two formats: the YAML files can be accessed directly through the GitHub interface or from a clone of the GitHub repository. Alternatively, the registry data can be accessed in XML format at the following URLs:

The following data... Can be accessed in XML format via URL...
Project https://topology.opensciencegrid.org/miscproject/xml
Resource Downtime https://topology.opensciencegrid.org/rgdowntime/xml
Resource Topology https://topology.opensciencegrid.org/rgsummary/xml
Virtual Organization https://topology.opensciencegrid.org/vosummary/xml

These XML pages are compatible with the XML format once provided by myosg.grid.iu.edu.

XML page arguments

Some of the XML pages listed above accept arguments for filtering queries. These are compatible with the arguments once used by myosg.grid.iu.edu.

For the Resource Topology (rgsummary) and Resource Downtime (rgdowntime) pages

Boolean filters:

The following argument(s)... Will have the effect of...
active=on&active_value=0 showing only inactive resources
active=on&active_value=1 showing only active resources
disable=on&disable_value=0 showing only disabled resources
disable=on&disable_value=1 showing only enabled resources
gridtype=on&gridtype_1=on showing only production resources
gridtype=on&gridtype_2=on showing only non-production (ITB) resources
service_hidden_value=0 showing only non-hidden services *
service_hidden_value=1 showing only hidden services *
has_wlcg=on showing only resources with WLCG information **

* Note lack of service_hidden=on argument
** There is no way to show only resources without WLCG information; has_wlcg=off doesn't work

ID-based filters:

These select by a numeric ID on facilities, sites, support centers, resource groups, and services, and VOs. Multiple IDs may be specified for an attribute, in which case resources that match any of the IDs listed will be shown. There are two equivalent formats for the arguments:

  • <ATTRIB>=on&<ATTRIB>_<ID1>=on&<ATTRIB>_<ID2>=on&<ATTRIB>_<ID3>=on
  • <ATTRIB>=on&<ATTRIB>_sel[]=<ID1>&<ATTRIB>_sel[]=<ID2>&<ATTRIB>_sel[]=<ID3>
The following attributes (<ATTRIB>)... Will filter by (<ID>)...
facility Facility ID
site Site ID
rg ResourceGroup GroupID
service Service ID
sc SupportCenter ID
voown VO Ownership (rgsummary only)

For example, either of the two following sets of arguments will show resources that are in ANL (10089), Fermilab (10009), or UW-Madison (10011):

  • facility=on&facility_10089=on&facility_10009=on&facility_10011=on
  • facility=on&facility_sel[]=10089&facility_sel[]=10009&facility_sel[]=10011

If multiple filters are specified, only resources that match all filters will be shown.

Finally, you may restrict how much past downtime is shown in the Resource Downtime XML with:

  • downtime_attrs_showpast=all: show all past downtime
  • downtime_attrs_showpast=<DAYS>: show downtime that ended no earlier than <DAYS> days ago. For example downtime_attrs_showpast=7 shows downtime that ended within the past 168 hours (7 days * 24 hours per day).
  • downtime_attrs_showpast=0: show no past downtime; this is the default if not specified

For the Virtual Organization (vosummary) page

Boolean filters:

The following argument(s)... Will have the effect of...
active=on&active_value=0 showing only inactive VOs
active=on&active_value=1 showing only active VOs
disable=on&disable_value=0 showing only disabled VOs
disable=on&disable_value=1 showing only enabled VOs
oasis=on&oasis_value=0 showing only VOs that do not use OASIS
oasis=on&oasis_value=1 showing only VOs that use OASIS

ID-based filter:

This selects based on a numeric ID on VOs. Multiple IDs may be specified, in which case VOs that match any of the IDs listed will be shown. There are two equivalent formats for the arguments:

  • vo=on&vo_<ID1>=on&vo_<ID2>=on&vo_<ID3>=on
  • vo=on&vo_sel[]=<ID1>&vo_sel[]=<ID2>&vo_sel[]=<ID3>

For example, either of the following will match GLOW (13), HCC (67), and IceCube (38):

  • vo=on&vo_13=on&vo_67=on&vo_38=on
  • vo=on&vo_sel[]=13&vo_sel[]=67&vo_sel[]=38

Getting Help

If you have any questions or encounter any issues with the registry, please open a ticket through the system appropriate for the VO(s) that you support:

If you primarily support... Submit new tickets to...
LHC VOs GGUS
Anyone else Helpdesk

Or email us at [email protected].

topology's People

Contributors

bbockelm avatar biozit avatar bockjoo avatar brianhlin avatar caffretro avatar cannonlock avatar carriebrown avatar christinalk avatar djw8605 avatar dougbenjamin avatar edquist avatar efajardo avatar hseverini avatar jdost321 avatar jeffreynpeterson avatar jlstephen avatar jmarra13 avatar juztas avatar lmichael107 avatar matyasselmeci avatar mmascher avatar mwestphall avatar osg-bot avatar ptmcguigan avatar rachellombardi avatar rynge avatar showmic09 avatar wuwj6269 avatar yzz971115 avatar zvada avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

topology's Issues

Cleanup VO translation

Looking here: https://github.com/opensciencegrid/topology/blob/master/virtual-organizations/CMS.yaml

Some thoughts:

  1. Remove redundant contact entries on conversion.
  2. Remove the contact IDs. We only need the IDs for cases where we can filter via the web.
  3. Remove information (such as the list of resources) that can be calculated in other places. Since the resource YAML will include the membership info, we don't also need it here too.
  4. Remove defaults entries.

This should drastically simplify the VO entries.

Filter VOs based on OASIS status

The filtering in MyOSG for OASIS attributes for VO queries is not replicated to the current VO responses.

We should implement this for OASIS attributes and the active attributes (if we don't already). I see we can additionally filter on support center ... it's not clear if that's necessary.

Badly named Perfsonar topology resource groups

(I accidentally first made this with no description)

I see a few topology/*/*/*erf*.yaml files that have names which have nothing to do with the site name. Sometimes those resource group names are just generically named with perfsonar in them:

topology/CBPF/GridLAFEX/PerfSONAR Services.yaml
topology/National Energy Research Scientific Computing Center/NERSC/Perfsonar Services.yaml
topology/Universidade de Sao Paulo/SAMPA/perfSONAR Services.yaml

Those should be renamed to have the facility name in there somehow.

Clean up GOC resources

There are quite a few resources associated with the GOC that should be either removed entirely if they won't have an effect on GRACC accounting (cc @djw8605 ) or deactivated and disabled.

Store all sources under src/ dir

To improve usability for our end-users, we should move all the sources under a src/ dir so that we have the following structure

projects/
src/
topology/
.gitignore
.travis.yml
virtual-organizations/
README.md
template-*.yaml

Suggestions for osg-notify

I have these suggestions to osg-notify. I sent them to @bbockelm by slack but I don't know if he noticed.

  1. I don't think that --recipients should be required if --oim-recipients is given
  2. I think --oim-recipients should have an option to do both resources and vos, not just either
  3. I think it would be helpful for the command to give a little feedback to the user upon success, such as the number of recipients that were sent the message.

Xenon Project "duplicated"

warning: the following paths have collided (e.g. case-sensitive paths
on a case-insensitive filesystem) and only one from the same
colliding group is in the working tree:

'projects/XENON.yaml'
'projects/xenon.yaml'

Add entries for services migrated from GOC

The following things are now hosted at University of Nebraska-Omaha:

  • topology server
  • osg-display
  • software repos

@djw8605 , anything else?

We should create entries for them. Use the new "Topology Server" service type for the topology server; the other services already have service type entries.

Add 'cms' project

There are many records that do not have a FieldOfScience field due to the lack of a cms project in topology.

@bbockelm asked me to create it, but I am not familiar with cms to input the correct values for a new project file.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.