The actual error system of Cozy Stack API is unusable to me.
They depends on 2 or 3 different system (cozy-stack, oauth, couchdb). Each has his response system / his response syntax.

OAuth:
http error: 400
request: doctype/_find
JSON response: {"error":"code=400, message=Expired token"}

CouchDB:
http error: 400
request: doctype/_find
JSON response: {"error":"no_usable_index","ok":false,"reason":"There is no index available for this selector.","status":"400"}

A request on cozy stack should have only one system of response to be usable by third party. I can't make generic request function on the stack with this.
This two error are made with the same code depends on the state of the oauth token. If it's an old token or a good one but with error on something else.

I can't use the HTTP Status Code to make a token refresh or display error to my user due to data he send or data i send.
Use the json as determination of error type is impossible due to the fact the error message can change during lifetime of the stack/couchdb/oauth and will break things if it's the case and it depends on which system is answering to my request.

I can't present usable message to my user too.

Needs for push notifications to be implemented

Way for the application to define certificates and tokens.

Both GCM (Google Cloud Messaging) and APNS (Apple Push Notification Service)
necessitate some kind of credentials. The application need to be registered
to the stack if it needs to use the push notification service.

• Apple
  • Apple Push Certificate.
  • BundleId
  • Passphrase (optional)
  • Production Mode
• Android
  • GCM sender id
  • GMC apiKey

Here is the necessary data on the Parse platform to configure push notifications.

push: {
  android: {
    senderId: '', // The Sender ID of GCM
    apiKey: '' // The Server API Key of GCM
  },
  ios: {
    pfx: '', // The filename of private key and certificate in PFX or PKCS12 format from disk  
    passphrase: '', // optional password to your p12
    bundleId: '', // The bundle identifier associated with your app
    production: false // Specifies which APNS environment to connect to: Production (if true) or Sandbox
  }
}

Source: http://docs.parseplatform.org/parse-server/guide/#push-notifications

The stack should add configuration options in the cozy.yml for this.

Way for a device to register itself

Devices need to be registered so we can send them push notifications.
A io.cozy.device doctype could be created. It would store the necessary device information to be able
to push notifications. It needs to be cleaned up if the push notifications fails too much.

Jobs to send notifications

A push attribute could be added to the notifications doctype that would send
the notification via mobile.

Way to bypass Google Services. microG is an open source implementation of the Google Services. It can be
used in place of Google Services but the phone needs to be rooted.

Source: https://github.com/microg/android_packages_apps_GmsCore

When creating a job via the HTTP API, the started_at seems wrong : 0001-01-01T00:00:00Z

Route : http://cozy.tools:8080/jobs/queue/konnector

Request :

{
    "data": {
        "attributes": {
            "options": {
                "priority": 10,
                "max_exec_count": 1
            },
            "arguments": {
                "konnector": "edf",
                "account": "372eba93e674f0b73037f05abd07fc9b",
                "folder_to_save": "372eba93e674f0b73037f05abd003feb"
            }
        }
    }
}

Response :

{
    "data": {
        "type": "io.cozy.jobs",
        "id": "372eba93e674f0b73037f05abd07fd32",
        "attributes": {
            "_id": "372eba93e674f0b73037f05abd07fd32",
            "_rev": "1-8487a4c23ac420dd3259577f6d439b67",
            "domain": "cozy.tools:8080",
            "worker": "konnector",
            "message": {
                "Data": "eyJrb25uZWN0b3IiOiJlZGYiLCJhY2NvdW50IjoiMzcyZWJhOTNlNjc0ZjBiNzMwMzdmMDVhYmQwN2ZjOWIiLCJmb2xkZXJfdG9fc2F2ZSI6IjM3MmViYTkzZTY3NGYwYjczMDM3ZjA1YWJkMDAzZmViIn0=",
                "Type": "json"
            },
            "options": {
                "max_exec_count": 1,
                "max_exec_time": 0,
                "timeout": 0
            },
            "state": "queued",
            "queued_at": "2017-05-16T19:13:54.144184557+02:00",
            "started_at": "0001-01-01T00:00:00Z",
            "error": ""
        },
        "meta": {
            "rev": "1-8487a4c23ac420dd3259577f6d439b67"
        },
        "links": {
            "self": "/jobs/konnector/372eba93e674f0b73037f05abd07fd32"
        }
    }
}

With cozy.create, it should be possible to add metadata to a file. For example it could be useful for konnectors to "label" files they are creating with the version of the konnector that created the file (for migration, backup, removing of old files).

When I do cozy-stack instances add --dev "cozy.local:8080" --apps drive,toto,settings,onboarding --passphrase cozy, I have no warning/error that the app toto is not found, despite the error emitted in the cozy-stack serve console (ERRO[0416] Failed to install toto: Unknown app domain="cozy.local:8080").

We use the secret download feature to allow users to open files in a new tab. But it looks like the stack sends headers that force the browser to download the file.

Hi

I'm having a problem with the instance domain and reverse proxy redirection, trying to install cozy-stack on debian Stretch with nginx as a reverse proxy.

I am using the latest binary release (cozy-stack-linux-amd64-2017M2-alpha) and couchdb in a docker (up to date) as in your guide.
I've been through the cozy-stack serve and curl -H 'Accept: application/json' 'http://cozy.tools:8080/status/' which does work as expected.

The problem comes when I want to access the cozy-stack from my my computer. I added an instance this way :
cozy-stack instances add --dev --apps drive,settings,onboarding --passphrase cozy "cozy.lvh.me:8081"
I chose lvh.me to stay consistent with my web services and chose the 8081 port because a node instance already is on the 8080.

[Note : To make this command work, I had to create a file ./.cozy/cozy-admin-passphrase with the command cozy-stack config passwd . (and cozy as a password) from my working directory with the cozy-stack binary (this isn't on the installation guide)]

I can then access the login page from my web browser (on my computer, outside the server). When I try to login with the password cozy, i get redirected to cozy.lvh.me:8081 !
Here is the http response :

HTTP/1.1 200 OK
Server: nginx
Date: Tue, 27 Jun 2017 20:57:30 GMT
Content-Type: application/json; charset=UTF-8
Content-Length: 45
Connection: keep-alive
Content-Security-Policy: default-src 'self';img-src *;
Set-Cookie: cozysessid=AAAAAFlSxro0YzdlZjlmNzg4ZGU1Yzk2MTEyNmJmZGRhNzAyZmZjM3wLyeWMDj8lsvC-v1CbNiBMcTCN5T00p2CmQ56j9-dM; Path=/; Domain=cozy.lvh.me; Max-Age=604800; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
{"redirect":"http://drive.cozy.lvh.me:8081/"}

The body is indeed {"redirect":"http://drive.cozy.lvh.me:8081/"} which cannot work. It seems that the cozy instance is trying to redirect me based on the local domain name.

How am i supposed to create the instance and configure my nginx reverse proxy ? How cna I configure the internet domain name to my cozy-stack ?

Here is my nginx config file :

server {
       listen         XXX.XXX.XXX.XXX:80;
       server_name    cozy.mydoma.in;

        root /var/www/nginx;
        index index.html;

        access_log /var/log/nginx/cozy_access.log;
        error_log /var/log/nginx/cozy_error.log;

        location / {
                proxy_pass http://cozy.lvh.me:8081;
        }
}

I tried instaciating the instance with my domain name :
./cozy-stack instances add --dev --apps drive,settings,onboarding --passphrase cozy "cozy.mydoma.in"
and chaging the nginx config file to proxy_pass http://cozy.mydoma.in; but of course that is not the solution, as I cannot access the service at all.

Hope I was clear

See

But folder is not mandatory in accounts. For example, MAIF accounts may not have a folder.

We are unable to run connector on account without a folder. Sad.

Nous avons besoin qu'un service soit lancé après le succès d'un autre service. Notre use case est : lorsque des transactions bancaires sont remontées par un connecteur, nous envoyons des emails d'alertes en cas de transactions importantes. Ces emails contiennent la catégorie d'un email. Nous cherchons maintenant à faire de la catégorisation automatique de transactions. Nous voulons donc que le service de catégorisation automatique se lance avant le service d'envoi d'email. Il me semble qu'il n'y a pas de moyen documenté de faire ca. Nous avions pensé à mettre une durée de debounce plus importante pour le deuxième service mais cela nous semble assez fragile.

Une deuxième solution serait d'avoir des évenéments déclenchés automatiquement à la fin d'un service : ex "service:success:categorisation" qui serait le trigger pour le service d'email. Qu'en pensez vous ?

Hi,
Just after a $ docker pull cozy/cozy-app-dev , my usual command-like :

$ docker run --rm -it \
    -p 8080:8080 \
    -v "$HOME/appone":/data/cozy-app/appone \
    -v "$HOME/apptwo":/data/cozy-app/apptwo \
    cozy/cozy-app-dev

shows the following message :

WARNING:
  No manifest.webapp file has been found in the mounted
  directory /data/cozy-app. The stack will be started
  without serving any local application.

and obviously, then, the apps aren't reachable in my browser.

But the single app way still works :

    -p 8080:8080 \
    -p 8025:8025 \
    -v "$HOME/myapp":/data/cozy-app \
    cozy/cozy-app-dev

Working on the interaction between the VFS and Couch, I have some suggestions to change the current API of our data layer, mainly on the couch.Db type. I'll try to list the different options that I've seen and the and the one I think is best.

I hope this is not something part of a previous discussion. I have not seen it.

Doc as a map[string]interface{}

The current one. While it's maybe the most direct and basic solution, and perfect for the lose /data api, it does not give us many possibilities for inner usages.

Working with map is not very idiomatic when we would like to enforce more type safety on the documents we store in database. And converting maps to structs is neither efficient nor desirable all over our code base.

Finally, map being implemented as a hashmap, it is less efficient than a packed struct.

Doc as a interface{}

By using the empty iface type, we have one level more of indirection. It allows us to use type structs or maps, which one is better to represent our data.

Although we have to rely on reflection and convention (an _id field on the struct for instance) to solve the "id problem". We do not gain much in type safety... while Go can provide us better possibilities by representing this behaviour through methods.

Doc as a interface { SetID, GetID }

Finally, we can enforce the existence of setter/getter on the type's methods to solve this issue. We of course still have one level on indirection more, but I think it is a useful abstraction in this case with better type safety and more idiomatic.

Regarding the rev, we can either enforce SetRev/GetRev methods on the doc types through the interface, or returning the rev at the return site of the CreateDoc func:

type Doc interface {
  GetID() string
  SetID(string id)
}

func CreateDoc(..., doc Doc) (rev string, error)

I think this last solution is better. See an implementation here.

What do you think ?

NB : This is a KB issue for discussion. To be closed and moved in the wiki/docs when we agree on the path forward

Existing packages

https://github.com/timjacobi/go-couchdb

last edit 2016-08
Apparently one of the firsts
Includes a http module, trying to figure out why, might simply be history (started in 2014)
fork of the "original" https://github.com/fjl/go-couchdb
brother of https://github.com/pokstad/go-couchdb which is used in this cool article : http://pokstad.com/2015/04/18/couchdb-and-go.html
Have tools to run as couchdbapp / couchdbdaemon (we wont use these)
Allow to pass a net/http.RoundTripper for fine transport tunning.
Pr merged with some fix for couchdb2, not sure if there is more issue

https://github.com/zemirco/couchdb

last edit 2016-08
No functions for changes API
Have a special interface for document {GetID(), GetRev()}

https://github.com/rhinoman/couchdb-go

last edit 2016-04
No functions for changes API
Have functions for managing users and roles
Have functions for proxying requests (downloads / uploads)
support json byte[] as well as interface{} for documents
Is clean and clear, but more complex because of more functions. We will need most those (users, proxy)

https://github.com/dustin/go-couch

last edit 2016-08
Simplest

Plan B - Make our own from net/http

If we go this way, it will be a good idea to look at the code of the other packages for each function, some pitfalls could be avoided :

ex:

• https://github.com/rhinoman/couchdb-go/blob/94e6ab663d5789615eb061b52ed2e67310bac13f/connection.go#L81
• Use custom Director & httputil.ReverseProxy for file download / upload

Considerations on how we will use it

(see the docs/architecture.md for more info)

• One stack instance communicate with one couchdb 2.x server or cluster. The couchdb server/cluster has a lot of databases (nb_users x nb_data_types). The pair stack+couch should handle as many active users as possible. Number of unactive users must not impact perfs too much.
  Note : definition of active user TBD (mobile app and 3rd party software are syncing using *dav and some background jobs like fetching banks accounts and threshold alerts )
• HTTPS between stack and couchdb is not a priority. If it's an option, it will allow more flexibility for ops team, but we can always find alternative solution to secure channel between stack and couchdb and self-hosted users will probably have both on an single machine.
• We wont keep changes feed open, as it was identified in couchdb workshop as the limiting factor on number of databases we can have and we want to have a bazillion databases. We will probably do some kind of polling, this will need to be investigated deeper.
• Binaries will NOT be stored in couchdb. We will only store in couchdb reference to the file in another storage solution (FS / Swift)
• The stack will use couchdb in 2 ways :
  A - Just proxying to/from the client, we will still need to parse the JSON, but will only read or changes a few special fields (_id, _type, _tags ....) for ACL and then pipe it to/from couchdb. We wont need to make sense of the whole document and can eventually get away without parsing it on some routes. This part should be totally flexible on the json content.
  B - Some APIs and Jobs will need to make sense of the given documents, so (un)marshal them from/into smarter struct (CalendarEvent) to be used in business logic.

Analysis

• They are all relatively similar in term of API. rhinoman has some more functions that we will need but lack changes feed.
• They all have a struct for database which hold configuration, as most of our request will be on different databases, we will have a great churn for these structure. This is less than optimal for RAM & GC, but probably insignificant against JSON parsing.
• They all use JSON Marshal et encode to accept any interface{} as couchdb doc.
• They are all relatively inactive / stable, not sure if it is because they are "finished" or abandonware.
• No library has a notion of pooling connection. This is handled at the net/http.Transport level in golang.
• Only the first library has option for fine-tunning of the internal Client/Transport,

Hi,
I followed the guide written by @clochix here: https://docs.cozy.io/en/install/manual/
Specifically, I created the cozy user without his home directory as explicitly mentioned in the guide:

adduser --system \
        --no-create-home \
        --shell /bin/bash \
        --group --gecos \
          "Cozy" cozy

Now, when I start cozy-serve with the command (from the guide):

sudo -b -u cozy sh -c '/usr/local/bin/cozy-stack serve \
     --log-level info \
     --host 0.0.0.0 >> /var/log/cozy/cozy.log 2>> /var/log/cozy/cozy-err.log'

I get the error of the title:

/var/log/cozy/cozy-err.log
::::::::::::::
time="2017-07-30T11:01:39+02:00" level=info msg="[jobs] trigger @event(d76482f7dd2f93cfb2b7a18955017001): Starting trigger" nspace=mem-scheduler
Error: mkdir /home/cozy: permission denied

And the server stops.
I tried (to check): sudo -b -u cozy sh -c ls and it works fine.
I checked my /etc/cozy/cozy.yaml file and it does not contain any mention to this directory. However the /etc/passwd mentions /home/cozy as the home dir of the cozy user:
cozy:x:126:136:Cozy,,,:/home/cozy:/bin/bash

Any idea of what's going wrong? Should I still create the /home/cozy directory?

I'm running the latest cozy on raspbian jessie (and I got cozy "correctly" working from the basic install guide)
(BTW, cozy-stack version outputs a blank line, maybe I should open an issue for this?)

Currently, Travis does not run go test and it would be nice.

I did not manage to find how to tell Travis to use my fork instead of original/upstream one

Sending build context to Docker daemon 2.048 kB
Step 1 : FROM golang
---> f69c27b2f59a
Step 2 : RUN go get github.com/cozy/cozy-stack
---> Running in 26412f110b09

github.com/cozy/cozy-stack/apps

src/github.com/cozy/cozy-stack/apps/git.go:89: cannot use gfs (type *gfs) as type "gopkg.in/src-d/go-git.v4/utils/fs".Filesystem in argument to filesystem.NewStorage:
*gfs does not implement "gopkg.in/src-d/go-git.v4/utils/fs".Filesystem (missing OpenFile method)
src/github.com/cozy/cozy-stack/apps/git.go:324: cannot use newGFS(fs.vfsC, fs.Join(fs.base, name)) (type *gfs) as type "gopkg.in/src-d/go-git.v4/utils/fs".Filesystem in return argument:
*gfs does not implement "gopkg.in/src-d/go-git.v4/utils/fs".Filesystem (missing OpenFile method)
src/github.com/cozy/cozy-stack/apps/git.go:333: cannot use gfs literal (type *gfs) as type "gopkg.in/src-d/go-git.v4/utils/fs".Filesystem in assignment:
*gfs does not implement "gopkg.in/src-d/go-git.v4/utils/fs".Filesystem (missing OpenFile method)

Hi,
Since a recent update of the stack in local, and since a few week in **.mycozy.cloud instances, I got request rejected by a 400 "Invalid JWT token" when I try to get images through /remote/com.omdbapi.img. Others call to the API and even other requests trough /remote still work. I can still make it work through an 2 month old cozy/cozy-app-dev docker image.

The request which fails dosen't use directly cozy-client-js, but directly an XmlHttpRequest, mimicking cozy-client-js source code, mainly :

Promise.all([
         cozy.client.authorize(),
         cozy.client.fullpath(`/remote/${doctype}?${options.params}`),
       ]).then((res) => {
         const xhr = new XMLHttpRequest();
         xhr.open('GET', res[1]);
         xhr.setRequestHeader('Authorization', res[0].token.toAuthHeader());

You can check it here : https://gitlab.cozycloud.cc/gjacquart/lamusiquedemesfilms/blob/master/app/lib/img_fetcher.js#L5-30

There are two ways to request documents in Couchdb2.

MapReduce doc

by PUTing a _design/xxxx document, with a map view, which can be arbiraty complex

// simple : sort all doc by name 
function(contact){
     emit(contact.name)
}

// more complex : tags
function(note){
    for(var i=0, l=note.tags.length; i<l; i++{
         emit(note.tags[i], note.title)
    }
}

// complex & arbitrary : get all done todo, sorted by hour of completion, 
// to see at what point of the day we are more efficient.
function(todo){
    if(todo.done){
         emit(new Date(todo.doneDate).getHours(), todo.complexity)
    }
}

This construct an index though the key (first argument of emit), which can they be requested with key=xx or sliced with startKey=xxxx... This returns the value (second argument of emit) or the full doc with include_docs=true.

This is very powerful, and allow for very complex query and statistics

Indexes & Mango doc

by POSTing an index

{
    "index": { "fields": ["firstname", "lastname"]  },
}

we can then request it using the mango query language

{
 "selector": {"firstname": "John"},
 "sort": ["lastname"],
 "limit": 2,
}

This is erlang powered and thus faster as there is no need to format the doc to JSON and pass it through IPC to the internal couchdb JS server.
However it is more limited : it is impossible to do manipulation of the data (like extracting hour from the datetime in the map exemple). And, more limiting, it is impossible to emit for each value of an array or map (tag exemple - couchdb isssue).

The vast majority of query from the previous stack and application can be made using the index&mango system : sort by field, get docs where fields is x, where field start with x.
A few query could not ( exemple , exemple2 )

My opinion is that we should focus on Indexes&Mango at first as it is faster, simpler and clearer for our firsts uses cases : get all files & folders by parentid. However we will need to support JS map reduce at some point in the future, but we should make it a less desirable option when proper client-side denormalisation of the document would allow to use indexes&mango (adding a completed_hour field in our task example)

On a fresh install of cozy-stack, triggers/io-cozy-triggers CouchDB database is missing and not automatically created, and so cozy-stack serve fails.

DEBU[0000] request: GET triggers%2Fio-cozy-triggers/_all_docs?include_docs=true&limit=1000   domain=triggers
DEBU[0000] CouchDB(not_found): Database does not exist.  domain=triggers

github.com/manyminds/api2go/jsonapi

The marshall/unmarshall uses an approach similar to JSON marshall/unmarshall in the standard lib, but it can be customized in many ways by implementing some interfaces. It looks like a low-level way to support JSON-API, with some code needed (not just a Struct description), but also the more extensible lib.

• Limitations: limited set of links (self, related and pagination), errors are in api2go package (not the jsonapi subset), not possible to customize self and related links in relationships
• 382 stars on github
• Last commit 27 days ago
• Extensions: building Rest API (with adapters for Gin and Gorilla)
• No active development, but seems well maintained

https://github.com/google/jsonapi

The jsonapi StructTags tells this library how to marshal and unmarshal your structs into JSON API payloads.

• Limitations: no support for top-level errors, no links for relationships, StructTags for relationships is far too limited
• Extensions: none
• 224 stars
• Last commit 5 days ago
• No active development, but seems maintained (even if a PR is waiting for a maintainer)

https://github.com/derekdowling/go-json-spec-handler

It works in two times: first, the document is unmarshalled to a generic struct (with Errors, Links, Data), and the attributes in data are just some Raw JSON, so they are unmarshalled later to a specific struct (User for example). This struct is just for the attributes, it doesn't have the ID. The attributes unmarshalling/marshalling is using the json tags, so it can be hard to reuse them for marshalling to JSON for CouchDB (no ID).

• Limitations: no one-to-one relationship, only self and related links
• Extensions: a JSONAPI client (jsc) and a web framework around goji2 (jsh-api)
• 26 stars on github
• Last commit 2 months ago
• The maintainer seems to merge pull requests in a timely manner, but didn't write code himself in the last 6 months

If I activate redis://localhost:6379/7 in cozy.yaml, realtime breaks for jobs, there is no communication between the stack and the browser.

The sender for now is [email protected] or [email protected]. It would be good if there was the slug of the cozy somewhere. [email protected] for example ?

Related conversation

patrick
2:49 PM
hello seb, ocmment vous avez su que l email qu a recu fabien provient d isabelledurand ?

sblaisot:
2:50 PM
c'était écrit dans le marc de café  (édité)
2:50 PM
en l'occurence avec To: "Isabelle Durand" <[email protected]>

patrick:
2:51 PM
mais on sait pas si l email provient du cozy isabelledurand.cozy.rocks non ?

sblaisot:
2:51 PM
non

sblaisot:
4:16 PM
cependant, les cozy sont maintenant (en V3) des entités virtuelles. un cozy n'envoie pas de mail. c'est la stack ou un worker qui envoie le mail
4:17 PM
coté infra, nous n'avons aucun moyen de savoir au sujet de quel cozy la stack envoie un mail. juste qu'elle a envoyé un mail
4:18 PM
une bonne pratique serait donc que le mail référence le cozy dont il est question, dans les en-têtes du mail ou dans le corps

patrick:
4:21 PM
yep carrement

Hello,
When I execute :

./cozy-stack instances add --dev --apps drive,settings,onboarding --passphrase cozy "cozy.tools:8080"

I have as result:

# cozy-stack instances side
Failed to create instance for domain cozy.tools:8080
Error: bad_request: Referer header required.

# cozy-stack serve side
ERRO[0004] [http] POST /instances CouchDB(bad_request): Referer header required.  nspace=http

I don't know my Cozy version because when I type ./cozy-stack version, I have no result. But I have clone this repo yesterday.

It would be cool if this would be possible

$ export COZY_STACK_CLI_DOMAIN='https://recette.cozy.rocks'
$ cozy-stack apps ls

Having to type --domain 'https://recette.cozy.rocks' is a bit inconvenient :)

State of the union

We currently have 4 context-like objects

• gin.Context is the request context as provided by gin framework and is passed to the various middlewares, it host informations about the http.Request and http.Response to the stack as well as errors bound to the request.
• vfs.Context is an object we made to pass both the dbPrefix and the afero.fs that are used by the vfs package to perform its operations
• instance.Instance is an object we made that represent a cozy instance, it contains the dbprefix and a "storageURL" which is passed to afero to create. It has a .getVFSContext method which is used to create a VFS context.
• couchdb dbPrefix is a string (but we might need to add host/auth later) which is used by the couchdb package

We expect to at least add

• cache.Location which will indicate for each instance where to store the memory cache (redis...)

Proposal

To avoid too much coupling but still keeping this simple, vfs.Context andcouchdb.DBprefix should be made into interfaces.

Instance can then implement these interface:

• we only pass a &Instance around in the normal stack flow
• we create TestVFSContext and TestPrefix to keep vfs and couchdb test package-scoped

Going further

I have a feeling the way we test gin.Handlers by recreating the router in every web/xxx_test.go files is weird, but I cant think of a better solution. Let's not dwell on it if we end up changing the routing framework.

Pour les notifications, j'ai dû mettre en place mailhog pour visualiser les mails envoyés.

J'ai dû demander sur #front comment faire : @enguerran a écrit une documentation ici : cozy/cozy-drive#382. Il pense et moi aussi que ce serait mieux de l'intégrer directement dans la documentation de la stack.

@nono et @jinroh vous en pensez quoi ?

I am in the cozy meetup to create a Konnector, and decided to go with google calendar.
It seems the authentification will be an issue to deploy and publish the konnector.
Would it be possible to integrate this authentification directly into the stack itself ? (Managing the secret token, and the per-client authorisation)
I heard that a mechanism was developed for MAIF and Orange, would it be possible to duplicate them for google services ?

Hi all,

I try to install an application on a fresh instance:

/usr/bin/cozy-stack apps install --domain myinstance.domain.tld collect git://github.com/cozy/cozy-collect.git#build
Error: could not parse event stream

And I have in logs (in debug level):

2017-08-11T17:53:47.540817+00:00 cozy-host cozy[25652]: time="2017-08-11T17:53:47Z" level=debug msg="request: GET myinstance-domain-tld%2Fio-cozy-settings/io.cozy.settings.instance " domain=myinstance.domain.tld
2017-08-11T17:53:47.543247+00:00 cozy-host cozy[25652]: time="2017-08-11T17:53:47Z" level=debug msg="response: {\"_id\":\"io.cozy.settings.instance\",\"_rev\":\"1-9defa783326f07a5c2791af897b360ab\",\"context\":\"cozy_beta\",\"email\":\"[email protected]\",\"public_name\":\"myinstance.domain.tld\",\"tracking\":\"false\"}" domain=myinstance.domain.tld

When fetching a folder's contents using GET /files/:file-id, the returned JSON should contain a relationships.meta.count property (we rely on it for cozy-drive's virtualized lists). But when the folder is empty, the relationships.meta object is empty.

To have hot reloading, you need to serve your JS with webpack-dev-server and CSP disallow that. Would it be possible to have a flag on the stack that disables them ?

Slightly related to #683 , but putting it here to keep the conversations separate: we both found it curious that /files/archive requires a collection of path's and won't work with id's. Since a lot of routes accept both, I think it would make sense that this one does too.

To be done

Work started in: https://github.com/tomsquest/cozy-stack

• Look in current directory for a config file
• Handle error when reading config file
• Create a Config struct containing the app configuration (see below)
• Add a config command to display the parsed configuration (eg. ./cozy-stack config)
• Inject Config where needed, eg. cmd/status.go needs the database url

Already done in master

• Parse command line argument with Cobra
• Read config file with Viper
• Integrate Cobra with Viper

The Config struct

This structure contains the final configured key-value pairs.

Goals:

• Be injected into code which depends on configuration
• Not having reference/depends the global spf13/viper in application code
• Level of indirection between the config flag and the real values (eg. renaming prod to production internally)

Pseudo-code for Config:

type Config struct {
    Mode mode 
    Database struct {
            Url string
        }
}

config := Config{
    Mode: Production(),
    Database: {
        Url: "http://db:5984",
    },
}

@nono @aeris preliminarily feedback appreciated

Should we update https://github.com/cozy/cozy-stack/blob/master/docs/INSTALL.md with infos to install redis on the server/computer ?

Like docker pull redis and docker run -p 6379:6379 redis…

I just found out that notifications have no creation_date, I think it would be useful, I am debugging a very strange bug and I think this information would be useful.

Storing files and directories in the same couch dabatase is mandatory to allow us faster queries (more batching, less roundtrips) and simpler document search.

However, it also has some implications regarding our current api. We have a relation doctype <-> database which makes it difficult for files and directories to live in the same database.

I list two possibilities, there can be other though..

1) Get rid of the doctype <-> database relation

We could remove the DocType() method from the couchdb.Doc interface to remove this relation, and add a "dbname" parameter to CRUD methods. ie:

CreateDoc(dbprefix string, *Doc) -> CreateDoc(dbprefix, dbname string, *Doc)

This puts the burden on the user of the couchdb package to handle document type, but is also more generic and allows to store different document on the same database.

2) Unique DocType and a new field

We could have a unique DocType for files and folders io.cozy.files. While this allow us to keep our current model, doctype <-> database, it would require either:

• to change quite heavily the current API to get rid of the notion of io.cozy.folders
• some dexterity to go from our public model to the inner one (also making the code base less clear).

What do you think ?

When the stacks fails to connect, its error contains the connection string to CouchDB which contains login:password.

time="2017-06-05T14:16:37Z" level=error msg="CouchDB(no_couch): could not create connection with the server - Put http://HOST:PASSWORD@SERVER:5984/recette-cozy-works%2Fio-cozy-konnectors/io.cozy.konnectors%2Fedf: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"

If people paste their server logs in a public place like a GitHub issue, their server is compromised.

brew install couchdb
brew services start couchdb
go get -u github.com/cozy/cozy-stack
cozy-stack serve

Runs smoothly, couchdb running and accepting requests, same for cozy-stack. I also edited /etc/hosts to map cozy.tools to 127.0.0.1

When i run : cozy-stack instances add --dev --apps files,settings,onboarding --passphrase cozy "cozy.tools:8080"
I get :

Cozy-stack (client) log:

ERRO[0000] Failed to create instance for domain cozy.tools:8080
ERRO[0000] bad_request: Referer header required.

Cozy-stack (server) log:

ERRO[0010] [http] POST /instances CouchDB(bad_request): Referer header required.
ERRO[0032] [http] GET / CouchDB(bad_request): Referer header required.

Couchdb debug log :

[Mon, 24 Apr 2017 09:28:18 GMT] [debug] [<0.130.0>] 'PUT' /global%2Finstances {1,1} from "127.0.0.1"
Headers: [{'Accept',"application/json"},
          {'Accept-Encoding',"gzip"},
          {'Content-Length',"0"},
          {'Host',"localhost:5984"},
          {'User-Agent',"Go-http-client/1.1"}]
[Mon, 24 Apr 2017 09:28:18 GMT] [debug] [<0.130.0>] OAuth Params: []
[Mon, 24 Apr 2017 09:28:19 GMT] [info] [<0.130.0>] 127.0.0.1 - - PUT /global%2Finstances 201
[Mon, 24 Apr 2017 09:28:19 GMT] [debug] [<0.132.0>] 'POST' /global%2Finstances/_index {1,1} from "127.0.0.1"
Headers: [{'Accept',"application/json"},
          {'Accept-Encoding',"gzip"},
          {'Content-Length',"50"},
          {'Content-Type',"application/json"},
          {'Host',"localhost:5984"},
          {'User-Agent',"Go-http-client/1.1"}]
[Mon, 24 Apr 2017 09:28:19 GMT] [debug] [<0.132.0>] OAuth Params: []
[Mon, 24 Apr 2017 09:28:19 GMT] [debug] [<0.132.0>] Minor error in HTTP request: {bad_request,
                                                  <<"Referer header required.">>}
[Mon, 24 Apr 2017 09:28:19 GMT] [debug] [<0.132.0>] Stacktrace: [{couch_httpd,validate_referer,1,
                                     [{file,"couch_httpd.erl"},{line,418}]},
                                 {couch_httpd_db,db_doc_req,3,
                                     [{file,"couch_httpd_db.erl"},{line,532}]},
                                 {couch_httpd_db,do_db_req,2,
                                     [{file,"couch_httpd_db.erl"},{line,234}]},
                                 {couch_httpd,handle_request_int,5,
                                     [{file,"couch_httpd.erl"},{line,318}]},
                                 {mochiweb_http,headers,5,
                                     [{file,"mochiweb_http.erl"},{line,94}]},
                                 {proc_lib,init_p_do_apply,3,
                                     [{file,"proc_lib.erl"},{line,247}]}]
[Mon, 24 Apr 2017 09:28:19 GMT] [info] [<0.132.0>] 127.0.0.1 - - POST /global%2Finstances/_index 400
[Mon, 24 Apr 2017 09:28:19 GMT] [debug] [<0.132.0>] httpd 400 error response:
 {"error":"bad_request","reason":"Referer header required."}

When fetching GET /files/:file-id with a folder ID, the JSON response includes a path attribute, and even the included files have this attribute. But when fetching the same route with a file ID, the path attribute is missing. Same problem with the POST /files/_find and GET /data/:doctype/:id/relationships/references routes. This is problematic, as we need the paths to, say, download an album as an archive.

En suivant la doc https://docs.cozy.io/fr/install/debian/ pour installer un cozy sur Ubuntu 16.04,

après avoir ajouté le repo de paquets debian pour Xenial,

arrivé à l'installation de cozy-stack

https://hyp.is/GhqXOss2EeeP8le7G6iY-g/docs.cozy.io/install/debian/

pas de bol :

E: Impossible de trouver le paquet cozy-stack

(il reste l'option de l'installation à partir de gitub)

Bonjour,

Après plusieurs tentatives d’installation de Gozy sur ma machine, je me demandais si vous aviez prévue la mise en place d'un Dockerfile (ou d'un sript bash) pour simplifier le déploiement de Gozy.

What we want?

The Cozy stack will be able to store the binaries on several providers: local filesystem, Swift from Open Stack, in memory (for tests), etc. The metadata are kept in CouchDB and the Rest API will be the same, no matter what the storage provider is. So, we need to define a Go interface for our storage providers.

github.com/blang/vfs

• Available storages: local FS, memory
• Available wrappers: readonly, mount, prefix
• 119 stars on github
• Last commit 1 month ago
• No open issues/PR
• No active development

github.com/spf13/afero

• Available storages: local FS, memory, SFTP
• Available wrappers: basepath, readonly, regexp, http, copy on write, cache on read
• 702 stars on github
• Last commit 9 days ago
• Some PRs are waiting that a maintainer looks at them, but globally it has regular work
• 2 other projects that rely on afero:
  • https://github.com/fcavani/aferoboltdb
  • https://github.com/ryansb/af3ro

github.com/thoas/gostorages

• Available storages: local FS and S3
• 20 stars on github
• Last commit 2 months ago
• No active development

Hi,
When I start the cozy-stack and try to add an instance, I have an unauthorized error.
The stack doesn't have couchdb admin logs.

How can I give those credentials to the stack ?

Thanks you

J'ai cru en lisant la documentation que le champ content était non obligatoire.
Mais si on ne l'ajoute pas ça fait une erreur ici :

If we already have an OpenID Connect or SAML IDP can Cozy be an relying party? Perhaps auto-enroll authenticated users if they successfully authenticate at the IDP?

Hi, Casbin is an authorization library that supports models like ACL, RBAC, ABAC.

Related to RBAC, Casbin has several advantages:

1. roles can be cascaded, aka roles can have roles.
2. support resource roles, so users have their roles and resource have their roles too. role = group here.
3. the permission assignments (or policy in Casbin's language) can be persisted in files or database (Gorm or Xorm or any popular DBs).

And you can even customize your own access control model, for example, mix RBAC and ABAC together by using roles and attributes at the same time. It's very flexible.

Casbin can provide more flexibility and security than the current permission system. So I think this project will benefit from Casbin. Because most of the code in https://github.com/cozy/cozy-stack/tree/master/pkg/permissions can be replaced by Casbin. And Casbin is more thoroughly tested against many test cases (like in our examples). This is only a glimpse:

testEnforce(t, e, "alice", "data1", "read", true)
testEnforce(t, e, "alice", "data1", "write", false)
testEnforce(t, e, "alice", "data2", "read", false)
testEnforce(t, e, "alice", "data2", "write", false)
testEnforce(t, e, "bob", "data1", "read", false)
testEnforce(t, e, "bob", "data1", "write", false)
testEnforce(t, e, "bob", "data2", "read", false)
testEnforce(t, e, "bob", "data2", "write", true)

BTW, we also use alice and bob for example:)

Let me know if there's any question:) Thanks.

Hi
I would like to generate tls certificates using letsencrypt (with certbot).
Where can I find a list of the subdomains used by cozy ?

Hi,

I was working on an app for Cozy (really a WIP) which would implement something like Calendly.

TLDR: It aims at letting you set up some public endpoints to give to others, so that they can request a time slot in your calendar. Typically, if you often schedule 30mins meetings with people, you would have an endpoint to request a slot for such a meeting. Others would choose the available time slots which best suits them and you will just have to confirm or refuse it. If you confirm it, it gets automatically in your calendar. To get a better idea, you can try this link, do not worry, it is not plugged to any of my calendars.

I was trying to write it full clientside and to avoid too much code migration when the new stack will be out. There are two main concerns about this app:

• Being able to serve a route publicly in the app (but this should be doable without concerns if I understood completely)
• Being able to perform reduction of data on the server side.

The second point is crucial for such an app. In the public route, you must have access to all events stored in the cozy, to filter out possible time slots and avoid conflicts. However, you absolutely do not want to fetch them locally, as it means an anonymous browser would handle all the events in the cozy. So this should be done server-side.

At the moment, I guess it can be done quite easily with CouchDB MapReduce. Basically, one would select all the events and perform reduction to only emit "conflicting time slots", that is the minimum information required, without any information about the events themselves.

As far as I understood, cozy-stack will drop MapReduce in favor of Mango queries. I may have missed something, but I did not find anything about a way to perform reduction on fetched data using Mango queries.

Thanks

Hi

I cannot sync my bills on my personnal cozy cloud.

I am trying to link some accounts to collect bills and contracts. So far I've tried Free, Free mobile and voyages-sncf.
When I try connecting by providing my (verified) credentials, I get the following error (for Free) :

An error occured

Unfortunately, something went wrong when trying to connect to Free. Please retry or contact the Cozy support on our forum.

From my Cozy logs :

# journalctl -fu cozy.service
juil. 01 15:14:55 mydoma.in cozy-stack[27545]: time="2017-07-01T15:14:55+02:00" level=error msg="[job] konnector/0: error while performing job ae9b405d4fb926875dfdf3b122027f64: fork/exec : no such file or directory" nspace=redis-job

From my nginx reverse proxy logs (if any use) :

# tail -f /var/log/nginx/cozy_acces.log
XX.XX.XX.XX - - [01/Jul/2017:16:25:52 +0200] "PUT /data/io.cozy.accounts/ae9b405d4fb926875dfdf3b1220203ce HTTP/2.0" 200 749 "https://collect.cozy.mydoma.in/" "Mozilla/5.0 (X11; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
XX.XX.XX.XX - - [01/Jul/2017:16:25:52 +0200] "POST /jobs/queue/konnector HTTP/2.0" 202 982 "https://collect.cozy.mydoma.in/" "Mozilla/5.0 (X11; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
XX.XX.XX.XX - - [01/Jul/2017:16:25:53 +0200] "OPTIONS /jobs/ae9b405d4fb926875dfdf3b1220282fd HTTP/2.0" 204 415 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
XX.XX.XX.XX - - [01/Jul/2017:16:25:53 +0200] "GET /jobs/ae9b405d4fb926875dfdf3b1220282fd HTTP/2.0" 200 1042 "https://collect.cozy.mydoma.in/" "Mozilla/5.0 (X11; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
XX.XX.XX.XX - - [01/Jul/2017:16:25:52 +0200] "PUT /data/io.cozy.accounts/ae9b405d4fb926875dfdf3b1220203ce HTTP/2.0" 200 749 "https://collect.cozy.mydoma.in/" "Mozilla/5.0 (X11; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
XX.XX.XX.XX - - [01/Jul/2017:16:25:52 +0200] "POST /jobs/queue/konnector HTTP/2.0" 202 982 "https://collect.cozy.mydoma.in/" "Mozilla/5.0 (X11; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
XX.XX.XX.XX - - [01/Jul/2017:16:25:53 +0200] "OPTIONS /jobs/ae9b405d4fb926875dfdf3b1220282fd HTTP/2.0" 204 415 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
XX.XX.XX.XX - - [01/Jul/2017:16:25:53 +0200] "GET /jobs/ae9b405d4fb926875dfdf3b1220282fd HTTP/2.0" 200 1042 "https://collect.cozy.mydoma.in/" "Mozilla/5.0 (X11; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"

Nothing on the nginx error logs.
My server firewall is disabled. I installed cozy more or less this way : https://clochix.github.io/gozy-docs/en/install/manual/ but with the couchdb docker from this guide : https://github.com/cozy/cozy-stack/blob/master/docs/docker.md

It seems that after issues with CouchDB, it is not possible to get a konnector in a good state.

$ ssh recette.int.cozycloud.cc  'cozy-stack --domain recette.cozy.works konnectors update edf'
16:17 patrick: time="2017-06-05T14:16:37Z" level=error msg="CouchDB(no_couch): could not create connection with the server - Put http://xxxxxxxxx@ha-couch-1-int:5984/recette-cozy-works%2Fio-cozy-konnectors/io.cozy.konnectors%2Fedf: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
$ ssh recette.int.cozycloud.cc  'cozy-stack --domain recette.cozy.works konnectors update edf'
16:17 patrick: time="2017-06-05T14:16:42Z" level=error msg="Application is not in valid state to perform this operation"
$ ssh recette.int.cozycloud.cc  'cozy-stack --domain recette.cozy.works konnectors update edf'
16:17 patrick: time="2017-06-05T14:16:45Z" level=error msg="Application is not in valid state to perform this operation"
$ ssh recette.int.cozycloud.cc  'cozy-stack --domain recette.cozy.works konnectors uninstall edf'
16:17 patrick: time="2017-06-05T14:16:52Z" level=error msg="Unqualified error: Application is not in valid state to perform this operation"

State of the union

The current state of the codebase and documentation is as follow

GET http://example.cozycloud.cc/data/io.cozy.events/111aaa111
{
"_id":"io.cozy.events/111aaa111"
}

which translates in couchdb

"_id":"io.cozy.events/111aaa111"

This solution has some advantages

1. No special field in couchdb and no need to change of the document on every /data request
2. Open to changes in how we store data : all ids are uniques, so if we finally want to put all documents in a single couchdb base (in data-proxies or in main stack), it wont change the external API.
3. The doctype is visible in both the URL, which allow some optimisation (discard body if doctype doesnt match permissions), and the document itself which simplify interfacing with JS frameworks (no need to manipulate a [type, map[string]string] tupple)

However, it cause a few issues :

1. ID means either 111aaa111 or io.cozy.event/111aaa111 depending on where you are in codebase, these could be alleviated by a convention. Either,
   a. ID and QID (qualified ID) or
   b. UUID and ID.
2. IDs have to be generated on go side following the convention doctype+UUID, this means it is impossible to simply replicate from another couchdb database, unless it respect the same convention.
3. Extracting the doctype from the QID requires string manipulation. This is inefficient.

Plan B - _type field

An alternative could be

GET http://example.cozycloud.cc/data/io.cozy.event/111aaa111
{
"_id":"111aaa111",
"_type":"io.cozy.events"
}

which would translate in couchdb

{
"_id":"111aaa111", 
"$x-cozy-type":"io.cozy.events"
}

Advantages

1. It is clear what ID is.
2. It is straightforward to extract the Type from a JSON doc.

Inconvenients

1. Need to parse and change the json on every /data request, we cant store the _type field in couchdb (couchdb doesnt accept fields which name starts with _)
2. IDs are not unique, there could be conflict among fixed id ( like io.cozy.folders/pictures vs io.example.irc_chans/pictures) if we attempt to merge our bases. This could be aleviated by forcing fixed id to include the doctype, but then we get weird URL /data/io.cozy.folders/io.cozy.folders.root)