Giter Club home page Giter Club logo

waffle_gcs's Introduction

Waffle GCS

Build Status

Google Cloud Storage for Waffle

What's Waffle?

Waffle (formerly Arc) is a file uploading library for Elixir. It's main goal is to provide "plug and play" file uploading and retrieval functionality for any storage provider (e.g. AWS S3, Google Cloud Storage, etc).

What's Waffle GCS?

Waffle GCS provides an integration between Waffle and Google Cloud Storage. It is (in my opinion) the spiritual successor to arc_gcs. If you want to easily upload and retrieve files using Google Cloud Storage as your provider, and you also use Waffle, then this library is for you.

What's different from arc_gcs?

The major three differences are:

  1. Transitions from Arc to Waffle.
  2. Uses the official Google Cloud API client for Elixir rather than constructing XML requests and sending them over HTTP.
  3. (In Progress) Implements the v4 URL signing process in addition to the existing v2 process.

Because Google now officially builds client libraries for Elixir, it is more maintainable to use those libraries rather than relying on the older XML API. The v2 URL signing process is also being deprecated in favor of the v4 process. Although Google will give plenty of advance notice if/when the v2 process is becoming unsupported, it's again more maintainable to use Google best practices to ensure future compatibility.

Installation

Add it to your mix dependencies:

defp deps do
  [
    {:waffle_gcs, "~> 0.2"}
  ]
end

Configuration

All configuration values are stored under the :waffle app key. E.g.

config :waffle,
  storage: Waffle.Storage.Google.CloudStorage,
  bucket: "gcs-bucket",
  storage_dir: "uploads/waffle"

Note: a valid bucket name is a required config. This can either be a hard-coded string (e.g. "gcs-bucket") or a system env tuple (e.g. {:system, "WAFFLE_BUCKET"}). You can also override this in your definition module (e.g. def bucket(), do: "my-bucket").

Authentication is done through Goth which requires credentials (https://github.com/peburrows/goth#installation).

Custom Token Generation

By default, the credentials provided to Goth will be used to generate tokens. If you have multiple sets of credentials in Goth or otherwise need more control over token generation, you can define your own module:

defmodule MyCredentials do
  @behaviour Waffle.Storage.Google.Token.Fetcher
  @impl Waffle.Storage.Google.Token.Fetcher
  def get_token(scopes) when is_list(scopes), do: get_token(Enum.join(scopes, " "))
  @impl Waffle.Storage.Google.Token.Fetcher
  def get_token(scope) when is_binary(scope) do
    {:ok, token} = Goth.Token.for_scope({"[email protected]", scope})
    token.token
  end
end

And configure it to use this new module instead of the default token generation:

config :waffle,
  storage: Waffle.Storage.Google.CloudStorage,
  bucket: "gcs-bucket-name",
  token_fetcher: MyCredentials

URL Signing

If your bucket/object permissions do not allow for public access, you will need to use signed URLs to give people access to the uploaded files. This is done by either passing signed: true to CloudStorage.url/4 or by setting it in the configs (config :waffle, signed: true). The default expiration time is one hour but can be changed by setting the :expires_in config/option value. The value is the number of seconds the URL should remain valid for after generation.

GCS object headers

You can specify custom object headers by defining gcs_object_headers/2 in your definition which returns keyword list or map. E.g.

def gcs_object_headers(_version, {_file, _scope}) do
  [contentType: "image/jpeg"]
end

The list of all the supported attributes can be found here: https://hexdocs.pm/google_api_storage/GoogleApi.Storage.V1.Model.Object.html.

GCS optional params

You can specify optional params by defining gcs_optional_params/2 in your definition, which returns keywords list, E.g:

def gcs_optional_params(_version, {_file, _scope}) do
  [predefinedAcl: "publicRead"]
end

It will be used as optional_params argument in gcs request. List of all supported attributes can be found here: https://hexdocs.pm/google_api_storage/GoogleApi.Storage.V1.Api.Objects.html#storage_objects_insert_simple/7

waffle_gcs's People

Stargazers

Peter Mueller avatar  avatar Timothy Vanderaerden avatar Amos Kibet avatar Fabian Carlos avatar Nelson Estevão avatar Boris Kuznetsov avatar Monte Brown avatar Łukasz Strzebińczyk avatar Klemen Sever avatar

Watchers

James Cloos avatar Peter Mueller avatar  avatar

waffle_gcs's Issues

Error coming from kolorahl's repo

Hi! I just updated from kolorahl's last available commit, and straight up I get this error:

** (FunctionClauseError) no function clause matching in Goth.Token.for_scope/2
    (goth 1.2.0) lib/goth/token.ex:70: Goth.Token.for_scope(%Backend.Product.Creator{__meta__: #Ecto.Schema.Metadata<:loaded, "creators">, admin_relationships: #Ecto.Association.NotLoaded<association :admin_relationships is not loaded>, admins: #Ecto.Association.NotLoaded<association :admins is not loaded>, author_relationships: #Ecto.Association.NotLoaded<association :author_relationships is not loaded>, bio: "Something something", courses: #Ecto.Association.NotLoaded<association :courses is not loaded>, cover: %{file_name: "a144a57c-70d6-4650-9f22-883da7864487_1616926811951984000.png", updated_at: ~N[2021-03-28 10:20:13]}, handle: "a_handle", id: "2c856398-5df5-46eb-9c94-a997095363ea", inserted_at: ~N[2021-03-26 10:41:37], name: "A Person", role: nil, subscribers: #Ecto.Association.NotLoaded<association :subscribers is not loaded>, updated_at: ~N[2021-03-28 10:20:13]}, nil)
    (waffle_gcs 0.1.1) lib/waffle/storage/google/token/default_fetcher.ex:6: Waffle.Storage.Google.Token.DefaultFetcher.get_token/1
    (waffle_gcs 0.1.1) lib/waffle/storage/google/cloud_storage.ex:83: Waffle.Storage.Google.CloudStorage.conn/1
    (waffle_gcs 0.1.1) lib/waffle/storage/google/cloud_storage.ex:47: Waffle.Storage.Google.CloudStorage.put/3
    (waffle 1.1.4) lib/waffle/actions/store.ex:130: Waffle.Actions.Store.put_version/3
    (elixir 1.11.3) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
    (elixir 1.11.3) lib/task/supervised.ex:35: Task.Supervised.reply/5
    (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Function: #Function<1.48286728/0 in Waffle.Actions.Store.async_put_version/3>
    Args: []
21:58:51.472 [erl_level=error crash_reason=function_clause domain=cowboy pid=<0.12669.0> ] <error> Ranch protocol #PID<0.12670.0> of listener BackendWeb.Endpoint.HTTP (connection #PID<0.12669.0>, stream id 1) terminated
an exception was raised:
    ** (FunctionClauseError) no function clause matching in Goth.Token.for_scope/2
        (goth 1.2.0) lib/goth/token.ex:70: Goth.Token.for_scope(%Backend.Product.Creator{__meta__: #Ecto.Schema.Metadata<:loaded, "creators">, admin_relationships: #Ecto.Association.NotLoaded<association :admin_relationships is not loaded>, admins: #Ecto.Association.NotLoaded<association :admins is not loaded>, author_relationships: #Ecto.Association.NotLoaded<association :author_relationships is not loaded>, bio: "Something something", courses: #Ecto.Association.NotLoaded<association :courses is not loaded>, cover: %{file_name: "a144a57c-70d6-4650-9f22-883da7864487_1616926811951984000.png", updated_at: ~N[2021-03-28 10:20:13]}, handle: "a_handle", id: "2c856398-5df5-46eb-9c94-a997095363ea", inserted_at: ~N[2021-03-26 10:41:37], name: "A Person", role: nil, subscribers: #Ecto.Association.NotLoaded<association :subscribers is not loaded>, updated_at: ~N[2021-03-28 10:20:13]}, nil)
        (waffle_gcs 0.1.1) lib/waffle/storage/google/token/default_fetcher.ex:6: Waffle.Storage.Google.Token.DefaultFetcher.get_token/1
        (waffle_gcs 0.1.1) lib/waffle/storage/google/cloud_storage.ex:83: Waffle.Storage.Google.CloudStorage.conn/1
        (waffle_gcs 0.1.1) lib/waffle/storage/google/cloud_storage.ex:47: Waffle.Storage.Google.CloudStorage.put/3
        (waffle 1.1.4) lib/waffle/actions/store.ex:130: Waffle.Actions.Store.put_version/3
        (elixir 1.11.3) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
        (elixir 1.11.3) lib/task/supervised.ex:35: Task.Supervised.reply/5
        (stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3

Is this to be expected?

Also, my waffle config uses Waffle.Storage.Google.CloudStorage instead of Waffle.Storage.Google as shown in the setup example. I a newbie to Elixir, but trying to read the code it seems to me that Google acts just as a namespace and not an actual module?

Thanks :)

Making the file public?

@almirsarajcic Hi!

I need my files to become public when uploading, which is consistent to performing the uploads like this:

        ) :: object_or_error
  defp insert(conn, bucket, name, {:file, path}, acl) do
    Objects.storage_objects_insert_simple(
      conn,
      bucket,
      "multipart",
      %Object{name: name, acl: acl},
      path,
      predefinedAcl: "publicRead"
    )
  end

Specifically, I am used to setting predefinedAcl as "publicRead". Can I get the same effect in some way with current implementation, or do I need to open a PR that will let me set predefinedAcl on a request like this?

New hex release?

Hi! Any chance to get a new release pushed to Hex? The current one depends on an old version of Waffle.

Signed URL v2 returning error when using service account

Signed URL v2 returning error when using service account

Description

I am trying to generate a signed url using waffle with waffle_gcs. I configured my project according to the goth migration guide, and enabled the signed parameter in my config. The credentials are being set by using a JSON key, associated to a :service_account. However, when I call the url/1 function from my uploader I get the following error:

** (MatchError) no match of right hand side value: :error
    (waffle_gcs 0.2.0) lib/waffle/storage/google/url_v2.ex:82: Waffle.Storage.Google.UrlV2.build_signed_url/3
    (fliipa 0.1.0) lib/fliipa/uploaders/report_uploader.ex:6: Fliipa.Uploaders.ReportUploader.url/3
    iex:6: (file)

It seems that the client_email and private_key are not being set on the Goth.Config. I managed to make a workaround by explicitly setting the parameters in the Application module by using:

Goth.Config.set(:client_email, credentials["client_email"])
Goth.Config.set(:private_key, credentials["private_key"])

However, I don't know if this is an intended behavior or is a library failure.

Environment

  • Elixir 1.16.2 / OTP 25.3.2.10
  • goth 1.4.3
  • waffle 1.1.8 / waffle_gcs 0.2.0

Token scope not working ...

I implemented a custom token fetcher, since we are running a multi-tenant application. The scope is not being passed to the conn() function, and hence not being passed to the custom fetcher function

Looking at the amazon code in waffle, seems like the meta field is of the format {file, scope}. The second part of the tuple should be passed to the token fetcher

I'll fork this repo and create a PR

Goth Deps Update

Hey,

it seems to me that the current doc of goth(1.3) no longer documents the version used by waffle_gcs (1.1).
Therefore the setup and usage of waffle_gcs is a bit frustrating.

Regards

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.