Giter Club home page Giter Club logo

fog-backblaze's Introduction

Fog::Backblaze

Integration library for gem fog and Backblaze B2 Cloud Storage

Installation

Add this line to your application's Gemfile:

gem 'fog-backblaze'

Or install it with gem:

gem install fog-backblaze

Usage

With CarrierWave

CarrierWave.configure do |config|
  config.fog_credentials = {
    provider: 'backblaze',
    b2_key_id: 'xxxx',
    b2_key_token: 'zzzxxxccc'
  }
  config.fog_directory = 'my-b2-bucket-name'
  config.fog_public = true
end

For now only public buckets are supported with CarrierWave

With fog library

require "fog/backblaze"

connection = Fog::Storage.new(
  provider: 'backblaze',

  # with one key (more secure)
  # get your API keys at https://secure.backblaze.com/app_keys.htm
  b2_key_id: 'xxxx',
  b2_key_token: 'zzzxxxccc'

  # full access to b2 account (less secure)
  b2_account_id: '123456',
  b2_account_token: 'aaaaabbbbbccccddddeeeeeffffff111112222223333',

  # optional, used to make some operations faster
  b2_bucket_name: 'app-test',
  b2_bucket_id: '6ec42006ec42006ec42',

  logger: Logger.new(STDOUT).tap {|l|
    l.formatter = proc {|severity, datetime, progname, msg|
      "#{severity.to_s[0]} - #{datetime.strftime("%T.%L")}: #{msg}\n"
    }
  },

  token_cache: 'file.txt'
)

Create bucket

connection.directories.create(key: 'my-b2-bucket', public: true)
# or
connection.put_bucket("my-b2-bucket", public: true)

Get Bucket

bucket = connection.directories.get('my-b2-bucket')
bucket.name # => 'my-b2-bucket'
bucket.bucket_id # => "2ee4e45855e60c1c8c4bbd48"
bucket.bucket_type # => "allPrivate"
bucket.cors_rules # => []

Delete bucket

connection.directories.new(key: 'my-b2-bucket').destroy
# or
connection.delete_bucket("my-b2-bucket")

Upload File

file = directory.files.create(
  key: 'example.html',
  body: File.open(__FILE__)
)
# or
connection.put_object("my-b2-bucket", "example.html", File.open(__FILE__))

List Files

directory.files.each do |file|
  p [directory.name, file.name]
end
# or
connection.list_objects("my-b2-bucket").json['files'] do |file_hash|
  p ["my-b2-bucket", file_hash['fileName']]
end

Read File

directory.files.new(key: 'example.html').body
# or
connection.get_object("my-b2-bucket", "example.html")

Get public URL

connection.get_public_object_url("my-b2-bucket", "example.html")

Delete File

directory.files.new(key: 'example.html').destroy
# or
connection.delete_object("my-b2-bucket", "example.html")

See example for more details

Adding b2_bucket_id

Most of internal operations requires bucketId field, to get right value, fog-backblaze will make API request. Usually applications use only one bucket and it's id never change (it may change only if we delete bucket and create new one with same name). We can eliminate this API request by setting b2_bucket_id attribute.

How to get b2_bucket_id:

p connection._get_bucket_id(bucket_name)

Token Cache

Each request requires authentication token, it comes from b2_authorize_account response.

Let's say we want to upload a files, then it will make 4 requests inernally:

  1. b2_authorize_account - valid for 24 hours
  2. b2_list_buckets - to get bucket_id value can be optimized with :b2_bucket_id field (should not change)
  3. b2_get_upload_url - valid for 24 hours
  4. Send data to URL from step 3

Results of steps 1, 2, 3 can be re-used by saving in TokenCache. It acts as general cachin interface with few predefined implementations:

  • In memory store token_cache: :memory (default)
  • JSON file store token_cache: 'file.txt'
  • Null store (will not cache anything) token_cache: false or token_cache: Fog::Backblaze::TokenCache::NullTokenCache.new
  • Create your custom, see token_cache.rb for examples

fog-backblaze's People

Contributors

bnauta avatar brycied00d avatar dan avatar gagoit avatar matthiaswinkelmann avatar okdas avatar paxa avatar plribeiro3000 avatar shanecav84 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

fog-backblaze's Issues

no implicit conversion of ActionDispatch::Http::UploadedFile into String

I seem to be getting the following error when submitting my form for my model:

no implicit conversion of ActionDispatch::Http::UploadedFile into String

Some details of my setup are below:

Rails 6.0.2.2
Ruby 2.6.0p0
gem 'fog-backblaze', '> 0.3.0’
gem 'shrine-fog', '
> 2.0', '>= 2.0.1’
gem 'shrine', '~> 3.2', '>= 3.2.1'

Model

class Show < ApplicationRecord
  include ImageUploader::Attachment(:artwork)
end

Image Uploader

class ImageUploader < Shrine
  plugin :store_dimensions
  
  Attacher.validate do
    validate_max_size  10*1024*1024
    validate_mime_type %w[image/jpeg image/png]
    validate_extension %w[jpg jpeg png]
  end
end

Shrine.rb

require "shrine"
require "shrine/storage/fog"
require "fog/backblaze"

storeb2 = Fog::Storage.new(
  provider: 'backblaze',
  b2_key_id: ‘###',
  b2_key_token: ‘###',
  b2_bucket_name: ’test',
  b2_bucket_id: ‘###'
)

Shrine.storages[:store] = Shrine::Storage::Fog.new(
  connection: storeb2,
  directory: “test",
)

Shrine.storages[:cache] = Shrine::Storage::Fog.new(
  connection: storeb2,
  directory: “test",
)
 
Shrine.plugin :activerecord 
Shrine.plugin :cached_attachment_data
Shrine.plugin :restore_cached_data 
Shrine.plugin :validation
Shrine.plugin :validation_helpers

Controller Code Where it is blowing up

def create
  @show = Show.new(show_params)

  if @show.save
    flash[:notice] = "The new show has been added to your account"
    redirect_to @show
  else
     render :new 
  end
end

private

def show_params
  params.require(:show).permit(:title, :artwork)
end

Do you know what this could be?

Deprecation Warnings

It seems that there's a newish depreciation warning with the latest Fog.

[fog][DEPRECATION] Unable to load Fog::Backblaze::Storage
[fog][DEPRECATION] The format Fog::Storage::Backblaze is deprecated

ArgumentError: When I use recreate_version!

my fog-b2.rb
CarrierWave.configure do |config|
config.fog_provider = 'fog/backblaze'

config.fog_credentials = {
provider: 'backblaze',
b2_key_id: Rails.application.credentials.dig(:backblaze, :b2_key_id),
b2_key_token: Rails.application.credentials.dig(:backblaze, :b2_key_token),
# optional, if you wanna see B2 requests in log
logger: Rails.logger,
b2_bucket_name: Rails.application.credentials.dig(:backblaze, :b2_bucket_name),
b2_bucket_id: Rails.application.credentials.dig(:backblaze, :b2_bucket_id)
}
config.fog_directory = 'mydirec'
config.asset_host = 'https://mycdn.site.com/file/path'
config.fog_public = false # true # default is true

#fog_attributes supports only: content_type, last_modified, content_disposition
#config.fog_attributes = { ... }

#this can help debugging in development
config.ignore_integrity_errors = false
config.ignore_processing_errors = false
config.ignore_download_errors = false
end

#CarrierWave.clean_cached_files!


When I exec
Model.avatar.recreate_versions!(:thumb)

ArgumentError: wrong number of arguments (given 3, expected 2)
.rbenv/versions/2.6.2/lib/ruby/gems/2.6.0/gems/fog-backblaze-0.2.0/lib/fog/storage/backblaze/real.rb:148:in `head_object'

Publish Gem?

It appears the gem has not been published yet. When do you plan to do that?

Could not find a valid gem 'fog-backblaze' (>= 0) in any repository

Thanks!

NoMethodError: save for Fog::Storage::Backblaze::File

I am building a Rails 4 Application with Carrierwave and Backblaze B2 via Fog.

Locally, with the Carrierwave Storage set to :file it works but apparently not on production using fog storage.. can u help me with this?

class MicropostsController < ApplicationController
  before_action :logged_in_user, only: [:create, :destroy]
  before_action :correct_user,   only: :destroy

  def create
    @micropost = current_user.microposts.build(micropost_params)
    if @micropost.save
      flash[:success] = "Micropost created!"
      redirect_to root_url
    else
      @feed_items = []
      render 'static_pages/home'
    end
  end

best regards,
ben

stream file upload

by looking at the source code I saw that this gem read complete file before doing the upload and I'm worried how much memory this will acquire for large files.

if content.is_a?(IO)
content = content.read
end

any otherway to improve this, like using stream/chunked upload?

Private fog setting with carrierwave doesn't return URL

CarrierWave.configure do |config|
  config.fog_credentials = {
    provider: 'backblaze',
    b2_account_id: ENV['BACKBLAZE_ACCOUNT_ID'],
    b2_account_token: ENV['BACKBLAZE_ACCOUNT_TOKEN']
  }
  config.fog_public = false
end
# Public
fog_public = true
object.file.url 
# https://f001.backblazeb2.com/file/bucket/uploads/file.jpg
# Private
With fog_public = false
object.file.url
# /uploads/file.jpg

CVE-2019-16779

Name: excon
Version: 0.68.0
Advisory: CVE-2019-16779
URL: GHSA-q58g-455p-8vw9
Title: Race condition when using persistent connections
Solution: upgrade to >= 0.71.0

Just a heads up that you may want to look into updating dependencies :)

Carrierwave: undefined method `copy_object' for #<Fog::Backblaze::Storage::Real>

When using CarrierWave, I get this error when uploading. The file successfully uploads, but it looks like Carrierwave is trying to cache the uploaded file on disk after the upload.

undefined method `copy_object' for #<Fog::Backblaze::Storage::Real>

carrierwave (2.1.0) lib/carrierwave/storage/fog.rb:453:in `copy_to'
carrierwave (2.1.0) lib/carrierwave/storage/fog.rb:340:in `store'
carrierwave (2.1.0) lib/carrierwave/storage/fog.rb:86:in `store!'
carrierwave (2.1.0) lib/carrierwave/uploader/store.rb:66:in `block in store!'
carrierwave (2.1.0) lib/carrierwave/uploader/callbacks.rb:15:in `with_callbacks'
carrierwave (2.1.0) lib/carrierwave/uploader/store.rb:65:in `store!'
carrierwave (2.1.0) lib/carrierwave/mounter.rb:105:in `each'
carrierwave (2.1.0) lib/carrierwave/mounter.rb:105:in `store!'
carrierwave (2.1.0) lib/carrierwave/mount.rb:401:in `store_document!'
...

When I monkey patch Fog::Backblaze::Storage::Real like so:

def copy_object(*args)
  p args
end

The output is:

["my-bucket", "uploads/tmp/filename", "my-bucket", "uploads/filename", {}]

"uploads/tmp/filename" looks to be a local path and "uploads/filename" looks to be the remote path in the bucket.

undefined method `parse'

I am trying to use the sample code

pp connection.put_object("fog-smoke-test", "my file", "THISISATESTFILE").json

but it keep throwing

NoMethodError: undefined method `parse' for Fog::JSON:Module
Did you mean?  parent
from ~/.rvm/gems/ruby-2.5.1/gems/fog-backblaze-0.1.1/lib/fog/backblaze/json_response.rb:12:in `json'
``

put_object fails to honour overridden extra_headers and content-disposition

In debugging why Backblaze wasn't honouring the content_disposition I was passing to put_object I noticed that its Hash#merge is taking in a Hash that's indexed by symbols (the {'foo': 'bar'} syntax) and merging in a hash that's indexed by strings ({'foo' => 'bar'}) and as a result, user-defined Content-Type doesn't override the default b2/x-auto

This behaviour can be seen in the logger.debug output from the request body where there's both :"Content-Type" and "Content-Type", then look at the response contentType of application-octet stream which indicates that b2/x-auto had been used.
You will also notice :Authorization and "Authorization" headers which is the result of the same mis-match between the use of the symbol :Authorization in put_object and the use of a string Hash index in b2_command.
(Formatting is my own to make the lines easier to read)

# Request body
{:body=>"-- Body 597185 bytes --", 
 :headers=>{:Authorization=>"<redacted>",
 :"Content-Type"=>"b2/x-auto",
 :"X-Bz-File-Name"=>"<redacted>",
 :"X-Bz-Content-Sha1"=>"857b96203ddfa5c50052c003a3db08d20cb2fc35",
 "Content-Type"=>"text/plain",
 "X-Bz-Info-src_last_modified_millis"=>1563222099031,
 "Authorization"=>"<redacted>"}}

# API response
{"accountId"=>"b8597a332ec1",
 "action"=>"upload",
 "bucketId"=>"<redacted>",
 "contentLength"=>597185,
 "contentSha1"=>"857b96203ddfa5c50052c003a3db08d20cb2fc35",
 "contentType"=>"application/octet-stream",
 "fileId"=>"<redacted>",
 "fileInfo"=>{"src_last_modified_millis"=>"1563222099031"},
 "fileName"=>"<redacted>",
 "uploadTimestamp"=>1563236520000}

headers: {
'Authorization': upload_url['authorizationToken'],
'Content-Type': 'b2/x-auto',
'X-Bz-File-Name': "#{_esc_file(file_path)}",
'X-Bz-Content-Sha1': Digest::SHA1.hexdigest(content)
}.merge(extra_headers)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.