Giter Club home page Giter Club logo

unpub's Introduction

Unpub

pub

Unpub is a self-hosted private Dart Pub server for Enterprise, with a simple web interface to search and view packages information.

Screenshots

Screenshot

Usage

Command Line

pub global activate unpub
unpub --database mongodb://localhost:27017/dart_pub # Replace this with production database uri

Unpub use mongodb as meta information store and file system as package(tarball) store by default.

Dart API is also available for further customization.

Dart API

import 'package:mongo_dart/mongo_dart.dart';
import 'package:unpub/unpub.dart' as unpub;

main(List<String> args) async {
  final db = Db('mongodb://localhost:27017/dart_pub');
  await db.open(); // make sure the MongoDB connection opened

  final app = unpub.App(
    metaStore: unpub.MongoStore(db),
    packageStore: unpub.FileStore('./unpub-packages'),
  );

  final server = await app.serve('0.0.0.0', 4000);
  print('Serving at http://${server.address.host}:${server.port}');
}

Options

Option Description Default
metaStore (Required) Meta information store -
packageStore (Required) Package(tarball) store -
upstream Upstream url https://pub.dev
googleapisProxy Http(s) proxy to call googleapis (to get uploader email) -
uploadValidator See Package validator -

Usage behind reverse-proxy

Using unpub behind reverse proxy(nginx or another), ensure you have necessary headers

proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

# Workaround for: 
# Asynchronous error HttpException: 
# Trying to set 'Transfer-Encoding: Chunked' on HTTP 1.0 headers
proxy_http_version 1.1;

Package validator

Naming conflicts is a common issue for private registry. A reasonable solution is to add prefix to reduce conflict probability.

With uploadValidator you could check if uploaded package is valid.

var app = unpub.App(
  // ...
  uploadValidator: (Map<String, dynamic> pubspec, String uploaderEmail) {
    // Only allow packages with some specified prefixes to be uploaded
    var prefix = 'my_awesome_prefix_';
    var name = pubspec['name'] as String;
    if (!name.startsWith(prefix)) {
      throw 'Package name should starts with $prefix';
    }

    // Also, you can check if uploader email is valid
    if (!uploaderEmail.endsWith('@your-company.com')) {
      throw 'Uploader email invalid';
    }
  }
);

Customize meta and package store

Unpub is designed to be extensible. It is quite easy to customize your own meta store and package store.

import 'package:unpub/unpub.dart' as unpub;

class MyAwesomeMetaStore extends unpub.MetaStore {
  // Implement methods of MetaStore abstract class
  // ...
}

class MyAwesomePackageStore extends unpub.PackageStore {
  // Implement methods of PackageStore abstract class
  // ...
}

// Then use it
var app = unpub.App(
  metaStore: MyAwesomeMetaStore(),
  packageStore: MyAwesomePackageStore(),
);

Available Package Stores

  1. unpub_aws: AWS S3 package store, maintained by @CleanCode.

Badges

URL Badge
/badge/v/{package_name} badge example badge example
/badge/d/{package_name} badge example

Alternatives

  • pub-dev: Source code of pub.dev, which should be deployed at Google Cloud Platform.
  • pub_server: An alpha version of pub server provided by Dart team.

Credits

License

MIT

unpub's People

Contributors

amondnet avatar clean-cole avatar cmengler avatar dellgreen avatar megatronking avatar pd4d10 avatar sispravnikov avatar talisk avatar vishal07051999 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

unpub's Issues

{"error":{"message":"RangeError (index): Index out of range: no indices are valid: 0"}}

有人遇到这个问题没
在Centos7上报错,在ubuntu上面是好的
Environment

pub version or flutter pub version:
Pub 2.7.0
flutter:
OS version:
CentOS-7-x86_64-Everything-2009
Are you using the Chinese community mirror or a corporate firewall?
yes

Problem

when exec flutter packages pub publish --server=http://192.168.70.128

Error in Client

{"error":{"message":"RangeError (index): Index out of range: no indices are valid: 0"}}

Error in server

`#0 _Uint8ArrayView.[] (dart:typed_data-patch/typed_data_patch.dart:3779:7)
#1 InputStream.readUint16 (package:archive/src/util/input_stream.dart:206:22)
#2 GZipDecoder._readHeader (package:archive/src/gzip_decoder.dart:89:29)
#3 GZipDecoder.decodeBuffer (package:archive/src/gzip_decoder.dart:26:5)
#4 GZipDecoder.decodeBytes (package:archive/src/gzip_decoder.dart:17:12)
#5 FileRepository.upload (file:///home/wangzhen/dev/pub_server/example/src/examples/file_repository.dart:67:34)

#6 CopyAndWriteRepository.upload (file:///home/wangzhen/dev/pub_server/example/src/examples/cow_repository.dart:104:36)
#7 ShelfPubServer._uploadSimple (package:pub_server/shelf_pubserver.dart:389:38)

#8 ShelfPubServer.requestHandler (package:pub_server/shelf_pubserver.dart:202:16)
#9 logRequests... (package:shelf/src/middleware/logger.dart:30:46)
#10 new Future.sync (dart:async/future.dart:224:31)
#11 logRequests.. (package:shelf/src/middleware/logger.dart:30:23)
#12 handleRequest (package:shelf/shelf_io.dart:101:29)
#13 serveRequests.. (package:shelf/shelf_io.dart:63:34)
#14 _rootRunUnary (dart:async/zone.dart:1134:38)
#15 _CustomZone.runUnary (dart:async/zone.dart:1031:19)
#16 _CustomZone.runUnaryGuarded (dart:async/zone.dart:933:7)
#17 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:338:11)
#18 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:265:7)
#19 _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:766:19)
#20 _StreamController._add (dart:async/stream_controller.dart:642:7)
#21 _StreamController.add (dart:async/stream_controller.dart:588:5)
#22 _HttpServer._handleRequest (dart:_http/http_impl.dart:2828:19)
#23 new _HttpConnection. (dart:_http/http_impl.dart:2586:19)
#24 _rootRunUnary (dart:async/zone.dart:1134:38)
#25 _CustomZone.runUnary (dart:async/zone.dart:1031:19)
#26 _CustomZone.runUnaryGuarded (dart:async/zone.dart:933:7)
#27 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:338:11)
#28 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:265:7)
#29 _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:766:19)
#30 _StreamController._add (dart:async/stream_controller.dart:642:7)
#31 _StreamController.add (dart:async/stream_controller.dart:588:5)
#32 _HttpParser._headersEnd (dart:_http/http_parser.dart:388:17)
#33 _HttpParser._doParse (dart:_http/http_parser.dart:703:15)
#34 _HttpParser._parse (dart:_http/http_parser.dart:320:7)
#35 _HttpParser._onData (dart:_http/http_parser.dart:812:5)
#36 _rootRunUnary (dart:async/zone.dart:1134:38)
#37 _CustomZone.runUnary (dart:async/zone.dart:1031:19)
#38 _CustomZone.runUnaryGuarded (dart:async/zone.dart:933:7)
#39 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:338:11)
#40 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:265:7)
#41 _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:766:19)
#42 _StreamController._add (dart:async/stream_controller.dart:642:7)
#43 _StreamController.add (dart:async/stream_controller.dart:588:5)
#44 _Socket._onData (dart:io-patch/socket_patch.dart:1831:41)
#45 _rootRunUnary (dart:async/zone.dart:1138:13)
#46 _CustomZone.runUnary (dart:async/zone.dart:1031:19)
#47 _CustomZone.runUnaryGuarded (dart:async/zone.dart:933:7)
#48 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:338:11)
#49 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:265:7)
#50 _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:766:19)
#51 _StreamController._add (dart:async/stream_controller.dart:642:7)
#52 _StreamController.add (dart:async/stream_controller.dart:588:5)
#53 new _RawSocket. (dart:io-patch/socket_patch.dart:1379:33)
#54 _NativeSocket.issueReadEvent.issue (dart:io-patch/socket_patch.dart:899:14)
#55 _microtaskLoop (dart:async/schedule_microtask.dart:43:21)
#56 _startMicrotaskLoop (dart:async/schedule_microtask.dart:52:5)
#57 _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:118:13)
#58 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:175:5)

2021-06-18T14:35:52.106984 0:00:00.001049 POST [302] /api/packages/versions/newUpload
2021-06-18 14:35:52.131507 INFO pubserver.shelf_pubserver Finish simple upload (error: RangeError (index): Index out of range: no indices are valid: 0).
2021-06-18T14:35:52.131426 0:00:00.000307 GET [400] /api/packages/versions/newUploadFinish?error=RangeError%20(index)%3A%20Index%20out%20of%20range%3A%20no%20indices%20are%20valid%3A%200`

执行 unpub --database mongodb: 报错

SocketException: OS Error: Connection refused, errno = 61, address = 10.***.**, port = 52157
#0      _ConnectionManager._connect (package:mongo_dart/src/network/connection_manager.dart:23:5)
<asynchronous suspension>
#1      _ConnectionManager.open.<anonymous closure> (package:mongo_dart/src/network/connection_manager.dart:58:14)
#2      Future.forEach.<anonymous closure> (dart:async/future.dart:492:26)
#3      Future.doWhile.<anonymous closure> (dart:async/future.dart:534:26)
#4      _RootZone.runUnaryGuarded (dart:async/zone.dart:1316:10)
#5      _RootZone.bindUnaryCallbackGuarded.<anonymous closure> (dart:async/zone.dart:1355:26)
#6      Future.doWhile (dart:async/future.dart:549:18)
#7      Future.forEach (dart:async/future.dart:490:12)
#8      _ConnectionManager.open (package:mongo_dart/src/network/connection_manager.dart:56:19)
#9      Db.open.<anonymous closure> (package:mongo_dart/src/database/db.dart:220:33)
#10     new Future.sync (dart:async/future.dart:224:31)
#11     Db.open (package:mongo_dart/src/database/db.dart:207:19)
#12     main (file:///Users/renshiqian/.pub-cache/hosted/pub.flutter-io.cn/unpub-1.2.1/bin/unpub.dart:28:23)
#13     _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:305:32)
#14     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:174:12)

Failing to publish to local unpub with error: missing authorization header pub finished with exit code 1

I was publishing successfully to my local unpub server on Mac Mini M1 chip, but suddenly (maybe after flutter upgrade but not sure) I am getting the error

missing authorization header
pub finished with exit code 1

Tried
flutter pub logout
flutter pub login

I login successfully but still when publishing i get this error. My local unpub is on an http server (not https), I tried to remove unpub from the bin folder and reactivate by

flutter pub global activate unpub

Still getting same error. Any solutions?

MongoDB ConnectionException: Could not connect to :27017

unpub % unpub --database mongodb:/121.5.238.137/:27017/dart_pub
Unhandled exception:
MongoDB ConnectionException: Could not connect to :27017
#0 _ConnectionManager.open (package:mongo_dart/src/network/connection_manager.dart:79:9)

#1 Db.open (package:mongo_dart/src/database/db.dart:324:32)
#2 main (file:///Users/dayuyu/.pub-cache/hosted/pub.flutter-io.cn/unpub-1.2.2/bin/unpub.dart:28:23)
#3 _startIsolate. (dart:isolate-patch/isolate_patch.dart:299:32)
#4 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)
dayuyu@MacBook-Pro-4 unpub %

Pre-Release version on pub.dev?

2.0.0 is 7 months old. Is the main branch in a good enough place to at least do a pre-release version up to pub.dev? Would be nice to remove my dependency_overrides.

CC @pd4d10

Security suggestions

What is the recommendation on securing a private pub server? Does pub support authentication via HTTP, for example?

We are planning to run a protected server for all developers around the world but would like to restrict access using a token or username/password.

fine grained access permissions

the 2.15 changes got me thinking about fine grain access control.

How would we feel about passing a user identifier to each of the meta and file store api calls?

The email addresses obtained from the access token probably makes sense.

If there is interest I might be able to find time to contribute the change.

Thoughts?

Enhancement: Cache/proxy upstream dependencies

To save/shared team enterprise internet bandwidth, it would be a nice feature (like in other programming languages) that already downloaded public packages are stored on the local enterprise unpub server rather then downloaded from the internet again.

However it appears that the way dart pub works its not super straight-forward as the version of the package is not included in the initial query that requests all versions. This means that the upstream connection to get an updated list of versions is still required to provide to dart pub tool, but the urls of the supplied json will need to be changed to point back to the local enterprise unpub server for downloads to ensure download request go thru the local server.

The download api does contain the required version, so when a following download request is received, download/cache and serve the package archive back to the client.

API Key based access

For those of us who want the server to be publicly available, but restrict access to specific people, is there a way to make access API key based?

Upload package archives to s3 buckets

Lots of companies use aws and might benefit from being able to store the packages in a s3 bucket instead of using the unpub server local filesystem

documentation on how to use it

Questions :

  1. how to upload our own package to a local server that uses MongoDB?
  2. how to download the package ?

please provide with a example

MongoDB connection string with authentication doesn't work

I tried using this package and supplied a connection string that includes my username and password.

E.g.: mongodb+srv://myusername:[email protected]/test?retryWrites=true&w=majority

I got an error saying that the port is not valid (where my password was interpreted as the port). How should I format my connection string and supply my credentials?

How to setup unpub as https server not http

Hello,

I have managed to setup unpub server on a machine on our LAN and published successfully packages to it. Now I want to do the same setup but as https not http, I can't find any references to this except this pull request
#3
which is not merged.

So is there a way / guide on how to setup unpub on https?

Enhancement: organise filestore packages in sub directory structure

As the amount of uploads increases, it would be nicer to store uploads in a hierarchical sub directory structure to make its easier to navigate and backup for administrators. Equally theoretically some older file systems have file/folder limits that in extreme cases could be exceeded if this project ever evolved into caching/proxying upstream downloads also.

One such structure inspired by the current api could be something like:

$baseDir/packages/$package/versions/$package-$version.tar.gz

It may be worth adding this as a new implementation of PackageStore class so that users of the api have the flexibility to decide between using the existing way to store uploads, or use the new way.

Failing publish new package. HTTP Error

Description

Cannot upload new package in my vps

Log

IO  : Retry #2 for POST http://localhost:4000/api/packages/versions/newUpload...
IO  : HTTP POST http://localhost:4000/api/packages/versions/newUpload
    | content-type: multipart/form-data; boundary=dart-http-boundary-LTvBaD6KtuMV-CecDYvS_TT-_a1cQjTGqc+qtr.F+pU+wSoT3Eu
    | user-agent: Dart pub 2.16.2
IO  : HTTP error:
    | SocketException: Connection refused (OS Error: Connection refused, errno = 61), address = localhost, port = 63438
    |
    | dart:_http                                                       _HttpClient.openUrl
    | package:http/src/io_client.dart 35:38                            IOClient.send
    | package:pub/src/http.dart 68:43                                  _PubHttpClient.send
    | package:http/retry.dart 109:33                                   RetryClient.send

Failing to publish packages to local pub server with error: DetailedApiRequestError(status: 400, message: No error details. HTTP status was: 400.)

I have a local pub server setup on our local LAN, using a self signed https certificate

When trying to publish, I tried the following

$unpub_auth login
(successful authorization)
$dart pub token add https://192.168.8.174:4000 (server with unpub)
$flutter pub publish --server https://192.168.8.174:4000

I get the following error

DetailedApiRequestError(status: 400, message: No error details. HTTP status was: 400.)
pub finished with exit code 1

This is the output of my flutter doctor

[✓] Flutter (Channel stable, 2.10.5, on macOS 11.3.1 20E241 darwin-x64, locale en-EG)
[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
[!] Xcode - develop for iOS and macOS (Xcode 12.5)
! Flutter recommends a minimum Xcode version of 13.
Download the latest version or update via the Mac App Store.
[✓] Chrome - develop for the web
[✓] Android Studio (version 2020.3)
[✓] VS Code (version 1.66.2)
[✓] Connected device (3 available)
[✓] HTTP Host Availability

! Doctor found issues in 1 category.

Any help is greatly appreciated

How to use it on client side

It's community is not that much strong. And on it's documentation there is nothing explanation about how to use it on client side

package | plugin obfuscation

This issue is not related to unpub

Since I am publishing the package at the enterprise level and sharing it with the client, I need to obfuscation the code
is there any way to achieve this 🙁 ?

MongoDB ConnectionException: Could not connect to localhost:27013

dayuyu@MacBook-Pro-4 unpub % unpub --database mongodb://localhost:27013/dart_pub
Unhandled exception:
MongoDB ConnectionException: Could not connect to localhost:27013
#0 _ConnectionManager.open (package:mongo_dart/src/network/connection_manager.dart:79:9)

#1 Db.open (package:mongo_dart/src/database/db.dart:324:32)
#2 main (file:///Users/dayuyu/.pub-cache/hosted/pub.flutter-io.cn/unpub-1.2.2/bin/unpub.dart:28:23)
#3 _startIsolate. (dart:isolate-patch/isolate_patch.dart:299:32)
#4 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)
dayuyu@MacBook-Pro-4 unpub %

errmsg: command count requires authentication?

GET /webapi/packages?size=15
Error thrown by handler.
{ok: 0.0, errmsg: command count requires authentication, code: 13, codeName: Unauthorized}
package:shelf/src/middleware/logger.dart 30:62 logRequests...

Asynchronous error

Asynchronous error
HttpException: Trying to set 'Transfer-Encoding: Chunked' on HTTP 1.0 headers

metastore.dart is exported but models.dart isnt

I'm not an expert in Dart so apologies in advance if i'm wrong, but given that metastore.dart is exported and uses classes from models.dart, then shouldn't models.dart be exported also, so that alternative metastore implementations can be used when using the unpub api?

How to add uploaders to existing published packages

I have a local unpub server setup with mongodb and managed to publish my packages to my local server. How can other developers on my team when required publish new versions of the same package without getting error that they are not an uploader for this package.

fold ==null

Invalid server response:
{"error":{"message":"NoSuchMethodError: The method 'fold' was called on null.\nReceiver: null\nTried calling: fold(Instance of '_CopyingBytesBuilder', Closure: (BytesBuilder, List) => BytesBuilder)"}}

try {
  var email = await _getUploaderEmail(req);

  var mediaType = MediaType.parse(req.headers['content-type']);

  var boundary = mediaType.parameters['boundary'];
  MimeMultipart fileData;

  await for (MimeMultipart part
      in req.read().transform(MimeMultipartTransformer(boundary))) {
    if (fileData != null) continue;
    fileData = part;
  }

Documentation on how to use

So looks like a useful tool however when I started reading the documentation it goes straight into details on how to use its internal api..

What I actually wanted to see was:

  1. what changes are required in my pubspec.yaml to reference the private repo.
  2. how to I publish to a private repo.

'String?' is nullable and 'Object' isn't.

../../../.pub-cache/hosted/pub.flutter-io.cn/unpub-2.0.0/lib/src/app.dart:113:48: Error: The argument type 'String?' can't be assigned to the parameter type 'Object' because 'String?' is nullable and 'Object' isn't.

  • 'Object' is from 'dart:core'.
    var server = await shelf_io.serve(handler, host, port);

dart version:Dart SDK version: 2.16.1 (stable) (Tue Feb 8 12:02:33 2022 +0100) on "macos_x64"

unpub --database mongodb://localhost:27017/dart_pub does'nt work

I've run unpub --database mongodb://localhost:27017/dart_pub but got a error

Unhandled exception:
MongoDB ConnectionException: Could not connect to localhost:27017
- SocketException: OS Error: Connection refused, errno = 61, address = localhost, port = 50298
#0      _ConnectionManager.open (package:mongo_dart/src/network/connection_manager.dart:93:9)
<asynchronous suspension>
#1      Db.open (package:mongo_dart/src/database/db.dart:442:7)
<asynchronous suspension>
#2      main (file:///Users/minhtran/.pub-cache/hosted/pub.dartlang.org/unpub-2.0.0/bin/unpub.dart:27:3)
<asynchronous suspension>

Authentication with tokens for 3rd party package repositories

Starting Dart 2.15 the command dart pub tokens add can be used to add authentication for use with 3rd party package repositories.

This is documented here:
https://github.com/dart-lang/pub/blob/master/doc/repository-spec-v2.md

Feel free to ping me if you have questions, also just ping me in #hackers-pub-dev- on https://github.com/flutter/flutter/wiki/Chat


This might break your existing authentication flow used in unpub, I don't think that was ever something we wanted to support.

Who's using unpub?

Unpub is developed to host private Dart packages for the Flutter team of ByteDance at first.

Then we realized that these efforts could probably help Dart/Flutter developers to land their private registry painlessly, so we made it open-source.

If your company/organization is using it, it is appreciated to put its name and logo here.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.