googleapis / docuploader Goto Github PK

This tool uploads documentation for publishing to googleapis.dev.

License: Apache License 2.0

Shell 43.03% Python 53.27% Dockerfile 3.69%

docuploader's Introduction

Google APIs

This repository contains the original interface definitions of public Google APIs that support both REST and gRPC protocols. Reading the original interface definitions can provide a better understanding of Google APIs and help you to utilize them more efficiently. You can also use these definitions with open source tools to generate client libraries, documentation, and other artifacts.

Building

Bazel

The recommended way to build the API client libraries is through Bazel >= 4.2.2.

First, install bazel.

To build all libraries:

bazel build //...

To test all libraries:

bazel test //...

To build one library in all languages:

bazel build //google/example/library/v1/...

To build the Java package for one library:

bazel build //google/example/library/v1:google-cloud-example-library-v1-java

Bazel packages exist in all the libraries for Java, Go, Python, Ruby, Node.js, PHP and C#.

Overview

Google APIs are typically deployed as API services that are hosted under different DNS names. One API service may implement multiple APIs and multiple versions of the same API.

Google APIs use Protocol Buffers version 3 (proto3) as their Interface Definition Language (IDL) to define the API interface and the structure of the payload messages. The same interface definition is used for both REST and RPC versions of the API, which can be accessed over different wire protocols.

There are several ways of accessing Google APIs:

JSON over HTTP: You can access all Google APIs directly using JSON over HTTP, using Google API client library or third-party API client libraries.
Protocol Buffers over gRPC: You can access Google APIs published in this repository through GRPC, which is a high-performance binary RPC protocol over HTTP/2. It offers many useful features, including request/response multiplex and full-duplex streaming.
Google Cloud Client Libraries: You can use these libraries to access Google Cloud APIs. They are based on gRPC for better performance and provide idiomatic client surface for better developer experience.

Discussions

This repo contains copies of Google API definitions and related files. For discussions or to raise issues about Google API client libraries, GRPC or Google Cloud Client Libraries please refer to the repos associated with each area.

Repository Structure

This repository uses a directory hierarchy that reflects the Google API product structure. In general, every API has its own root directory, and each major version of the API has its own subdirectory. The proto package names exactly match the directory: this makes it easy to locate the proto definitions and ensures that the generated client libraries have idiomatic namespaces in most programming languages. Alongside the API directories live the configuration files for the GAPIC toolkit.

NOTE: The major version of an API is used to indicate breaking change to the API.

Generate gRPC Source Code

To generate gRPC source code for Google APIs in this repository, you first need to install both Protocol Buffers and gRPC on your local machine, then you can run make LANGUAGE=xxx all to generate the source code. You need to integrate the generated source code into your application build system.

NOTE: The Makefile is only intended to generate source code for the entire repository. It is not for generating linkable client library for a specific API. Please see other repositories under https://github.com/googleapis for generating linkable client libraries.

Go gRPC Source Code

It is difficult to generate Go gRPC source code from this repository, since Go has different directory structure. Please use this repository instead.

docuploader's People

Contributors

Stargazers

Watchers

Forkers

jskeet renovate-bot isabella232 tryweirder coryan qpc-github

docuploader's Issues

BUG: Failed docs jobs should not exit with status 0.

@tmatsuo reported that a failed docs job exited with status 0, making it appear the job had succeeded.

================================================================
Detecting version Fri Sep 13 07:32:42 PDT 2019.
================================================================
Creating docs metadata Fri Sep 13 07:32:42 PDT 2019.
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/kbuilder/.local/lib/python3.6/site-packages/docuploader/__main__.py", line 26, in <module>
    from docuploader.protos import metadata_pb2
  File "/home/kbuilder/.local/lib/python3.6/site-packages/docuploader/protos/metadata_pb2.py", line 28, in <module>
    dependencies=[google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR],
TypeError: __new__() got an unexpected keyword argument 'serialized_options'

Warning: a recent release failed

The following release PRs may have failed:

Investigate presubmit failure

Taking a look at the most recent presubmit failure:

nox > flake8 docuploader tests
nox > mypy docuploader
docuploader/shell.py:24: error: Incompatible default for argument "cwd" (default has type "None", argument has type "str")  [assignment]
docuploader/shell.py:24: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
docuploader/shell.py:24: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
docuploader/shell.py:27: error: Incompatible default for argument "timeout" (default has type "None", argument has type "int")  [assignment]
docuploader/shell.py:27: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
docuploader/shell.py:27: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
Found 2 errors in 1 file (checked 9 source files)
nox > Command mypy docuploader failed with exit code 1
nox > Session lint failed.

Not sure what changed between that could cause this... will look into fixing it.

Warning: a recent release failed

The following release PRs may have failed:

#102 - The release job failed -- check the build log.

Warning: a recent release failed

The following release PRs may have failed:

Warning: a recent release failed

The following release PRs may have failed:

#102 - The release job failed -- check the build log.

Configure Owlbot to ignore proto files

We should configure Owlbot to ignore proto files, and other generated files, as they do not require any post-processing.

Dependency Dashboard

This issue provides visibility into Renovate updates and their statuses. Learn more

This repository currently has no open or pending branches.

Check this box to trigger a request for Renovate to run again on this repository

Allow docs.metadata.json as an alternative to docs.metadata

It can be useful to create docs metadata from code rather than via the docuploader command line. Currently docuploader uses a textproto file format - if you could pass in a JSON file as an alternative (which could then just be parsed to the same metadata - or even just a separate command line to convert from JSON to textproto) that would make life a lot simpler in some cases.

Add timeout feature to run function in shell.py

Currently there is no way to add a timeout option to run function that is included in shell.py. This optional argument exists in subprocess library as of Python 3.3 which should also be utilized by the shell function in this docuploader

Warning: a recent release failed

The release PR #29 is still in a pending state after several hours

Docpublisher should store library metadata on the stem document

Some of the metadata applies to the entire library, not a specific version.

distribution_name
product_page
github_repository
issue_tracker

We can store those metadata fields on the document (cpp.google-cloud-firestore, for example) and use it to populate metadata fields for previous versions that didn't have those metadata fields.

Ensure `latest` is identical to the latest version docs

Currently when a docset for an existing version is updated, latest is not updated. We should explore other ways to keep latest in sync with the true latest version docset.

Docuploader should accept .repo-metadata.json format

In addition to docs.metadata, docuploader should be able to get the information it needs from .repo-metadata.json.

example .repo-metadata.json

Add ability to use Application Default Credentials

For example, when running docuplaoder from Cloud Build, we should pick up the Cloud Build service account.

Issue tracker URL should default to github issue tracker

We should be able to figure this out from the github repo metadata.

tar.py uses "--force-local" option on MacOSX, which doesn't exist

Environment details

OS: MacOSX
Language runtime version:
Package version: 0.6.2

Steps to reproduce

Run docuploader upload on MacOSX

$  docuploader upload out --staging-bucket docs-staging-v2-dev --destination-prefix docfx- --metadata-file /path/to/docs.metadata

Stack trace

  tar: Option --force-local is not supported                                                                                                                                                                                            
  Usage:                                                                                                                                                                                                                                
    List:    tar -tf <archive-filename>                                                                                                                                                                                                 
    Extract: tar -xf <archive-filename>                                                                                                                                                                                                 
    Create:  tar -cf <archive-filename> [filenames...]                                                                                                                                                                                  
    Help:    tar --help                                                                                                                                                                                                                 
                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                        
  Error Output:                                                                                                                                                                                                                         
  ================                                                                                                                                                                                                                      
  docuploader > Let's upload some docs!                                                                                                                                                                                   
  docuploader > Loading up your metadata.                                                                                                                                                                                 
  docuploader > Looks like we're uploading vision version 2.2.3 for php.                                                                                                                                                  
  docuploader > Sit tight, I'm tarring up your docs in out.                                                                                                                                                               
  docuploader > Failed executing tar --create --directory=out --file=/var/folders/l2/600rtyfd7bjb5k5ypp_g6xl8009z5m/T/tmpnyh50hxv --force-local --gzip --verbose .:                                                           
                                                                                                                                                                                                                                        
  None                                                                                                                                                                                                                              
  Traceback (most recent call last):                                                                                                                                                                                                    
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/click/core.py", line 1130, in __call__                                                                                                                         
      return self.main(*args, **kwargs)                                                                                                                                                                                                 
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/click/core.py", line 1055, in main                                                                                                                             
      rv = self.invoke(ctx)                                                                                                                                                                                                             
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/click/core.py", line 1657, in invoke                                                                                                                           
      return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                           
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/click/core.py", line 1404, in invoke                                                                                                                           
      return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                    
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/click/core.py", line 760, in invoke                                                                                                                            
      return __callback(*args, **kwargs)                                                                                                                                                                                                
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/docuploader/__main__.py", line 126, in upload                                                                                                                  
      docuploader.tar.compress(documentation_path, tar_filename)                                                                                                                                                                        
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/docuploader/tar.py", line 27, in compress                                                                                                                      
      return shell.run(                                                                                                                                                                                                                 
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/docuploader/shell.py", line 48, in run                                                                                                                         
      raise exc                                                                                                                                                                                                                         
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/docuploader/shell.py", line 35, in run                                                                                                                         
      return subprocess.run(                                                                                                                                                                                                            
    File "/opt/homebrew/Cellar/[email protected]/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run                                                                                                
      raise CalledProcessError(retcode, process.args,                                                                                                                                                                                   
  subprocess.CalledProcessError: Command '['tar', '--create', '--directory=out', '--file=/var/folders/l2/600rtyfd7bjb5k5ypp_g6xl8009z5m/T/tmpnyh50hxv', '--force-local', '--gzip', '--verbose', '.']' returned non-zero exit status 1.

Solution

The error is "tar: Option --force-local is not supported"

I was able to solve this by opening up tar.py and commenting out the option --force-local.

Update storage pin to v2

trying to update storage requirements on doc-templates repo: googleapis/doc-templates#337, but cannot be done unless storage version is bumped here as well, if possible.

Error "unsupported operand type(s) for -: 'tuple' and 'float'" when uploading

This is in a release build which has worked many times today - then failed for the last 6 builds. My guess is that an initial request failed, causing it to trigger some retry code which is broken. All builds are failing with this error:

Traceback (most recent call last):
  File "C:\Program Files (x86)\Python37-32\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Program Files (x86)\Python37-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\docuploader\__main__.py", line 176, in <module>
    main()
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\click\core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\docuploader\__main__.py", line 125, in upload
    credentials_file=credentials,
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\docuploader\upload.py", line 34, in upload
    blob.upload_from_filename(filename=source)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\google\cloud\storage\blob.py", line 1318, in upload_from_filename
    predefined_acl=predefined_acl,
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\google\cloud\storage\blob.py", line 1263, in upload_from_file
    client, file_obj, content_type, size, num_retries, predefined_acl
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\google\cloud\storage\blob.py", line 1169, in _do_upload
    client, stream, content_type, size, num_retries, predefined_acl
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\google\cloud\storage\blob.py", line 963, in _do_multipart_upload
    response = upload.transmit(transport, data, object_metadata, content_type)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\google\resumable_media\requests\upload.py", line 106, in transmit
    retry_strategy=self._retry_strategy,
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\google\resumable_media\requests\_helpers.py", line 136, in http_request
    return _helpers.wait_and_retry(func, RequestsMixin._get_status_code, retry_strategy)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\google\resumable_media\_helpers.py", line 150, in wait_and_retry
    response = func()
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\google\auth\transport\requests.py", line 252, in request
    self.credentials.before_request(auth_request, method, url, request_headers)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\google\auth\transport\requests.py", line 89, in __exit__
    self.remaining_timeout = self._timeout - elapsed
TypeError: unsupported operand type(s) for -: 'tuple' and 'float'

Add CI build without trampoline that runs create-metadata command

@busunkim96 Thanks! The code looks good. How about to have another ci build without trampoline and run at least create-metadata command so that we know it runs on the default kokoro image. I think we can do that in another PR.

Originally posted by @tmatsuo in #21 (comment)

Issue when using relative path for "metadata-file" option (and file is not in cwd)

The metadata-file option does not work as expected, and fails unless the file is in the CWD or an absolute path is used.

Environment details

Programming language: PHP
OS: MacOSX
Language runtime version:
Package version: 0.6.2

Steps to reproduce

Put docs.metadata file in the CWD and run the command
docuploader upload out --staging-bucket docs-staging-v2-dev --destination-prefix docfx-
This works as expected
Now put docs.metadata in the out directory and specify the path but running:
docuploader upload out --staging-bucket docs-staging-v2-dev --destination-prefix docfx- --metadata-file out/docs.metadata
This results in the following error: FileNotFoundError: [Errno 2] No such file or directory: 'out/out/docs.metadata' (full stacktrace below)
Now put JUST docs.metadata as the value for --metadata-file (maybe it expects the path to be relative to the output dir). This results in the following error: docuploader > You need metadata to upload the docs. You can generate it with docuploader create-metadata
Now supply the ABSOLUTE path to the option, OR move the docs.metadata file into the CWD and run with --metadata-file docs.metadata. That works as expected

Stack trace

docuploader upload out --staging-bucket docs-staging-v2-dev --destination-prefix docfx- --metadata-file out/docs.metadata 
docuploader > Let's upload some docs!
docuploader > Loading up your metadata.
Traceback (most recent call last):
  File "/Users/betterbrent/bin/docuploader", line 8, in <module>
    sys.exit(main())
  File "/Users/betterbrent/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/betterbrent/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/betterbrent/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/betterbrent/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/betterbrent/Library/Python/3.8/lib/python/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/betterbrent/Library/Python/3.8/lib/python/site-packages/docuploader/__main__.py", line 95, in upload
    shutil.copy(metadata_path, pathlib.Path(documentation_path) / metadata_file)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/shutil.py", line 418, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: 'out/out/docs.metadata'

Make sure prefix doesn't lead to double dash

If you specify a destination-prefix that ends in a -, docuploader adds another - after it, leading to something like docfx--my-package.

docuploader should only add a - if there isn't one already.

docuploader/docuploader/__main__.py

Lines 131 to 132 in 9433252

 if destination_prefix: 

 destination_name = f"{destination_prefix}-{destination_name}"

Do not upload tarball if directory is empty

Currently we accept documentation to be tar'd and uploaded even if there is nothing to be uploaded. This results in blobs that only contain a tarball with docs.metadata(\.json)? file, rather than throwing an error.

Relevant code snippet:

docuploader/docuploader/__main__.py

Lines 123 to 145 in 78511e7

 with tempfile.NamedTemporaryFile() as fh: 

 fh.close() 

 tar_filename = fh.name 

 docuploader.tar.compress(documentation_path, tar_filename) 

 docuploader.log.success("Cool, I have those tarred up.") 

 docuploader.log.info("Okay, I'm sending them to the cloud™ now.") 

 destination_name = ( 

 f"{metadata.language}-{metadata.name}-{metadata.version}.tar.gz" 

 ) 

 if destination_prefix: 

 # destination_prefix should end in alphanumeric character 

 while not destination_prefix[-1].isalnum(): 

 destination_prefix = destination_prefix[:-1] 

 destination_name = f"{destination_prefix}-{destination_name}" 

 docuploader.upload.upload( 

 source=tar_filename, 

 destination=destination_name, 

 bucket=staging_bucket, 

 credentials=credentials, 

 project_id=project_id, 

 )

Warning: a recent release failed

The following release PRs may have failed:

Release new version

To pick up changes from #26.

@busunkim96, would you be able to help with this?

Would like to be able to specify which tar command to use

I've discovered why --force-local didn't cause a problem on my Windows box, but did on Kokoro: I've got two (or more!) versions of tar installed. The one in c:\Windows\System32 doesn't need or support --force-local; the one elsewhere in my path requires --force-local in order to support filenames with colons in. Sigh.

I'm looking into workarounds for this (changing the path environment variable doesn't appear to help right now) but it would be nice if we could specify which tar command to use via an environment variable or something similar.

Fix failing lint sessions

Since August 22 the default py3.8 session for lint has been failing with

ox > Command python -m pip install mypy flake8 black types-pkg_resources failed with exit code 1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/google/home/dandhlee/workspace/docuploader/.nox/lint/lib/python3.8/site-packages/pip/__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "/usr/local/google/home/dandhlee/workspace/docuploader/.nox/lint/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/usr/local/google/home/dandhlee/workspace/docuploader/.nox/lint/lib/python3.8/site-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/usr/local/google/home/dandhlee/workspace/docuploader/.nox/lint/lib/python3.8/site-packages/pip/_internal/cli/main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "/usr/local/google/home/dandhlee/workspace/docuploader/.nox/lint/lib/python3.8/site-packages/pip/_internal/cli/cmdoptions.py", line 23, in <module>
    from pip._internal.cli.parser import ConfigOptionParser
  File "/usr/local/google/home/dandhlee/workspace/docuploader/.nox/lint/lib/python3.8/site-packages/pip/_internal/cli/parser.py", line 12, in <module>
    from pip._internal.configuration import Configuration, ConfigurationError
  File "/usr/local/google/home/dandhlee/workspace/docuploader/.nox/lint/lib/python3.8/site-packages/pip/_internal/configuration.py", line 27, in <module>
    from pip._internal.utils.misc import ensure_dir, enum
  File "/usr/local/google/home/dandhlee/workspace/docuploader/.nox/lint/lib/python3.8/site-packages/pip/_internal/utils/misc.py", line 42, in <module>
    from pip._internal.locations import get_major_minor_version, site_packages, user_site
  File "/usr/local/google/home/dandhlee/workspace/docuploader/.nox/lint/lib/python3.8/site-packages/pip/_internal/locations/__init__.py", line 14, in <module>
    from . import _distutils, _sysconfig
  File "/usr/local/google/home/dandhlee/workspace/docuploader/.nox/lint/lib/python3.8/site-packages/pip/_internal/locations/_distutils.py", line 9, in <module>
    from distutils.cmd import Command as DistutilsCommand
ModuleNotFoundError: No module named 'distutils.cmd'

not sure how this happened, but doesn't seem like it will resolve itself.

Write a README

@eaball35 mentioned an actual README would have been very helpful!

Lint failing due to missing mypy types

nox > mypy docuploader
docuploader/__main__.py:24: error: Library stubs not installed for "pkg_resources" (or incompatible with Python 3.6)
docuploader/__main__.py:24: note: Hint: "python3 -m pip install types-pkg_resources"
docuploader/__main__.py:24: note: (or run "mypy --install-types" to install all missing stub packages)
docuploader/__main__.py:24: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 9 source files)

Update README to use main branch when updated

There's a reference to master branch for google-cloud-go, once it is updated to the main branch we should update the link here as well.

Your .repo-metadata.json file has a problem 🤒

You have a problem with your .repo-metadata.json file:

Result of scan 📈:

must have required property 'library_type' in .repo-metadata.json
release_level must be equal to one of the allowed values in .repo-metadata.json

☝️ Once you address these problems, you can close this issue.

Need help?

Schema definition: lists valid options for each field.
API index: for gRPC libraries api_shortname should match the subdomain of an API's hostName.
Reach out to go/github-automation if you have any questions.

Require valid serving path when uploading Devsite content

You run docuploader create-metadata to create a docs.metadata file. That file contains metadata about the docs you're uploading. There is an option right now for --serving-path. When you call docuploader upload, it reads docs.metadata to figure out what the filename should be. At that point, I'd like to verify: if this is Devsite content, there is a serving path that matches the regex /{metadata.language}/docs/reference/[^/]+/[^/]+/[^/]+.

We need to do it during upload instead of create-metadata because at the point of create-metadata, we don't know if it's Devsite content because the --destination-prefix docfx flag is for upload.

We should start with a warning to avoid breaking existing builds. But, eventually we should make it an error.

FR: display date version was released

In the info popup, it would be nice to see what date the version was published/released. It would give some confidence that the docs being viewed are up-to-date.

Docuploader should accept github-repository in both formats

Use the full URL if provided https://github.com/googleapis/google-cloud-dotnet.

If just the repository name (googleapis/google-cloud-dotnet) is provided, construct the correct url.

Warning: a recent release failed

The following release PRs may have failed:

#102

Docpublisher should have a job to update /latest for all docs

This job would run after a publish-all.

Warning: a recent release failed

The following release PRs may have failed:

Consider documenting the file format docuploader creates

That would allow other tools to be written without Python dependencies etc.

Downsides:

Multiple implementations could be inconsistent
Implementations would need to be aware of changes

Warning: a recent release failed

The following release PRs may have failed:

#102 - The release job is 'autorelease: tagged', but expected 'autorelease: published'.

BUG: incomplete refdocs (Ruby, Java, maybe others?)

What I did:

Navigate to:

(Ruby) https://googleapis.dev/ruby/google-cloud-pubsub/latest/Google/Cloud/PubSub.html

(Java) https://googleapis.dev/java/google-cloud-clients/latest/com/google/cloud/pubsub/v1/Subscriber.Builder.html

What I saw

(Ruby) Unlinked references to Google::Cloud::Config

(Java) Unlinked references to com.google.api.gax.core.CredentialsProvider etc

What I expect

Able to click through to the docs for Cloud::Config, Java's gax, and any other docs for a dependency or transitive dependency.

Warning: a recent release failed

The following release PRs may have failed:

#102

Failure on the default kokoro image

An example log
https://source.cloud.google.com/results/invocations/aac0b577-2b30-4a9f-9f85-a62df6df5ac8/log

I think the current docuploader needs protobuf 3.6.0 or higher.

Docs that shouldn't exist are available on /latest

This is a weird one, sorry if it's hard to explain, or my fault somehow. In the google-api-nodejs-client, we have a customer reporting a bug with a doc that shouldn't exist:
googleapis/google-api-nodejs-client#1932

The issue is this collection of files/folder here:
https://googleapis.dev/nodejs/googleapis/latest/sqladmin/index.html

This API used to exist, but was removed. You can see here, no sqladmin:
https://github.com/googleapis/google-api-nodejs-client/tree/master/src/apis

I ran npm run docs to double check, and can confirm the sqladmin directory does not exist. So it feels like it may be possible we have the ghost of previous files from old versions sitting around and being served. Happy to sit with someone and debug this....

👻 👻 👻 👻 👻 👻 👻 👻 👻 👻 👻 👻

	if destination_prefix:
	destination_name = f"{destination_prefix}-{destination_name}"

	with tempfile.NamedTemporaryFile() as fh:
	fh.close()
	tar_filename = fh.name
	docuploader.tar.compress(documentation_path, tar_filename)
	docuploader.log.success("Cool, I have those tarred up.")
	docuploader.log.info("Okay, I'm sending them to the cloud™ now.")
	destination_name = (
	f"{metadata.language}-{metadata.name}-{metadata.version}.tar.gz"
	)
	if destination_prefix:
	# destination_prefix should end in alphanumeric character
	while not destination_prefix[-1].isalnum():
	destination_prefix = destination_prefix[:-1]

	destination_name = f"{destination_prefix}-{destination_name}"

	docuploader.upload.upload(
	source=tar_filename,
	destination=destination_name,
	bucket=staging_bucket,
	credentials=credentials,
	project_id=project_id,
	)

googleapis / docuploader Goto Github PK

docuploader's Introduction

Google APIs

Building

Bazel

Overview

Discussions

Repository Structure

Generate gRPC Source Code

Go gRPC Source Code

docuploader's People

Contributors

Stargazers

Watchers

Forkers

docuploader's Issues

Environment details

Steps to reproduce

Stack trace

Solution

Environment details

Steps to reproduce

Stack trace

Need help?

Recommend Projects

Recommend Topics

Recommend Org