Separate the GUI frontend, and allow Duplicati to run as a service about duplicati HOT 39 CLOSED

From [email protected] on March 13, 2009 08:34:07

Issue 25 has been merged into this issue.

from duplicati.

From [email protected] on March 20, 2009 03:13:23

I target the service/client splitup for version 2.0.

Labels: Milestone-Release2.0

from duplicati.

From [email protected] on July 08, 2009 12:14:09

I think Duplicati is great; but I am waiting for it to run as a service before
deploying it for mission-critical backups.

Looking forward to v2.0 - keep up the great work.

from duplicati.

From [email protected] on December 28, 2009 07:53:42

A possible solution would be to start Duplicati using screen. For example:

screen -dmS Duplicati ./StartDuplicati.sh

Or integrate the screen command in the skript file.

from duplicati.

From [email protected] on December 28, 2009 11:35:37

Can you re-connect the X session to an app started in a screen?

Anyway, good idea, but screen requires that the user logs in, and then runs the script.

To be fully useful, the service should be able to start without any users logged in,
and not require any userinterface. And I am not aware of a screen implementation for
windows.

from duplicati.

From jhiswin on April 21, 2010 10:35:49

The way I deal with this now is to use CommandLine and a scheduler.
It would be nice if CommandLine could use the GUI backup profiles.

Then you would have what would be effectively a running service.

from duplicati.

From [email protected] on April 24, 2010 01:19:18

I haven't thought of it that way, but yes, I can see where this could be usefull.
The downside to that solution is that you will rely on the properties of the scheduler to
see and control status.

For this issue I am planning to make this entirely integrated, so the UI looks close to
what it does now, but transmits/receives data from the service.

The timeline for the completion of this issue is a bit far ahead, if the current
development speed is an indicator, as there are still numerous issues awaiting 1.2.

The implementation of commandline/gui profile combination can be developed with
significantly less effort. If you would like such a feature, please create a ticket for
it, and we can discuss what it should look like.

from duplicati.

From [email protected] on October 12, 2010 12:42:38

Issue 281 has been merged into this issue.

from duplicati.

From [email protected] on October 15, 2010 08:19:52

Wouldn't it be possible to add Duplicati.CommandLine.exe via Windows API or Sc.exe [1] as a service during the installation?

It would require the addition of reading and respecting the scheduling data from the user's (or system's) profile, but that functionality appears to be implemented already in the GUI, so you could perhaps reuse that code.

[1] http://support.microsoft.com/kb/251192

from duplicati.

From [email protected] on October 15, 2010 08:48:07

Yes, that is a bit of a shortcut. Simply have a micro scheduler that reads the database and runs the commandline version at the right moments.

The service would have to accept some sort of control instructions though, so the user can stop/pause/throttle a backup. There is also a need for a reverse communication, so the user can see what backup is currently running, how far along it is etc.

It would also require that the GUI can access the sqlite database with the settings. One request is that it is possible to install a "headless" version, which only installs the service, and then be able to configure and inspect the backup from a remote location.

from duplicati.

From [email protected] on November 09, 2010 03:41:06

What is the best way of contributing to getting this resolved ? I don't have a lot of time to put in, but I am quite interested in seeing this progress.

from duplicati.

From [email protected] on November 09, 2010 11:08:14

I have an off-line project with some other people who should be working on a proposal for this. So some progress may be on the way.

The current state of the code is that the scheduler and backup process runs completely without being tied to the GUI. It could in theory just not create any UI elements and run as a service. (But that would not be manageable).

My expected approach is outlined below. If you have comments to a particular item, the sequence, overall approach, or particular use case, please let me know. If you feel comfortable tackling one of the tasks, you can get a sandbox and work on a branch of the current trunk.

1. Formats
   I would start out by identifying all requests being sent to the scheduler/runner, and define a format for those requests (preferably xml). Then I would identify all events being raised by the scheduler/runner, and define a format for those. This should be fairly easy.

2. Communication method
   Next step is to define a flexible method for sending requests and events between the two programs. This is slightly more complicated, because there are multiple factors to optimize for. A simple first version could just write files in a pre-agreed folder. This is inefficient and causes disk-wear so should not be the final version. I was thinking that it could be implemented as a simple http server, but that does not do two-way communication well, and polling would not be nice. But a http interface would simplify some later development, so I would prefer that.

3. Authentication
   When the communication channel and messages are well defined, an authentication system must be defined. It should be flexible enough to be portable and still allow a local user a login-free connection (perhaps using the GUI process credentials?).

I'm not entirely sure how the users are to be verified. I would prefer that the users are somehow extracted from the OS, so there is no local database to maintain. But that may be hard to make in a cross-platform compatible manner.

4. Multi-user
   Next, the service must be able to start multiple instances, each in the correct user context, so multiple users can use the same machine without interfering with each other.

5. Portable support
   The --portable-mode can be implemented simply by hosting the service in-process, and communicating internally with it through an alternate channel.

6. Remote display/editing
   If the communication channel already uses http, it should be very easy to make a small html page that requests the status and shows it in a browser.

If the backups should be entirely editable, there are some issues with the custom dialog components from backends, as they are Windows.Forms only. A first version could present the options in a grid-like fashion, a second version could include a html version of the UI.

7. VSS and other security stuff
   As the service will possibly run in the Administrator/root account, some extra options may be required to prevent non-privileged users to backup data they are not allowed to.

from duplicati.

From [email protected] on March 25, 2011 10:47:09

I just want to add my vote for this. I don't particularly care about all the API stuff, sending custom requests to the server etc. - I'd just like the icon to be gone from the task bar, so my users can't accidentally close Duplicati, and be able to control Duplicate via a web interface remotely.

I'm confused about point (3) and the need for a database. Isn't there already a database where the server stores the backup jobs it's supposed to do?

from duplicati.

From [email protected] on March 26, 2011 05:03:12

Yes, there is a database with the setup, but the idea is that Duplicati becomes manageable from a remote location, and since you probably do not want everyone to monitor/edit your backups remotely, there needs to be a list of approved logins/users.

I have not yet decided on how to manage this on a system with multiple users; should they share the backup database, or have one each? If they have one each, how do I figure out which one the (remote) user wants? Since this is still not decided, I would start of by making a user-login database.

from duplicati.

From jhiswin on March 26, 2011 07:05:10

I would like to recommend not integrating this directly into the main program.

There are other ways of remote management, such as RDP/VNC/SSH, which will likely already be in place. I also feel the same about the scheduler. Many systems already have a system in place for scheduling. If this is added as a feature, I would suggest making a separate program for creating scheduling rules, or a separate optional scheduling daemon.
Personally, I'd much prefer adding cron rules for backup myself, or creating Windows scheduled tasks myself. Allows me to better organize scheduled tasks, and I have plenty of flexibility and control.
There are also reporting and logging tools for monitoring backups. I could just create modules to parse and input logging data that Duplicati generates.

What I'm trying to say is that, scheduling and remote management would be separate programs in of itself.

from duplicati.

From [email protected] on April 01, 2011 02:04:51

As I see it, the current problem is that Duplicati cannot both run as a service/scheduled AND show a UI.

You can currently setup Duplicati to run with the system scheduler (cron/windows scheduler), and that option will not go away. But if you do so, you cannot pause Duplicati, should you want to.

For this to work, Duplicati will need to run as a separate process and accept commands from another process, which implies some form of inter-process communication.

It would be (almost) zero extra work to allow the communication channel to accept input from another machine, as opposed to limit it to local machine input.

Should you choose to only run the UI locally, you can still use VNC/RDP to view and manipulate the UI. This solution works for many cases, but if you are an IT administrator, you do not want to do this for, say 20 machines each day, to monitor backup status. If the inter-process communication allows external input, it is a matter of writing a program that queries all known hosts and presents the results in a sensible manner.

So I do not think there is any incentive for not making remote management a possibility. This will not remove the option of running the UI locally and using RDP/VNC, nor will it remove the option of running it "portable" or simply scheduled.

Should you wish not to use the remote management feature, it is a simple case of not opening the port in the firewall (possibly locking the listener in the program).

As for running it with the system scheduler, I can see some cases where this would be beneficial. For now I would prefer not to depend on the OS scheduler, as I can see some issues where the user modifies/deletes the schedule causing Duplicati and the scheduler to be out of sync, e.g. leaving un-used backups in Duplicati.

Feel free to add an issue requesting integration with the OS scheduler though, and we can discuss what should happen when.

from duplicati.

From jhiswin on April 01, 2011 06:18:42

So, something like xml or telnet-ish for remote management right?
I was thinking more along the lines of shell scripting or install package type management, but that makes sense.
Ya, you could just have it bind only to 127.0.0.1 or reject other IPs. And authentication too.

I personally don't like programs that take over scheduling, because it tends to spread out configuration all over the place when I have to manage things. But I guess it could be limiting for the backups.

For logging I was thinking more along the lines of things like Nagios, AWstats, and/or eventlog exporting.

So, would the plan be to daemonize the main program, and just make UI/cli interact using the remote management API? OS scheduler integration wouldn't really be necessary then as anyone needing that could easily just use the cli and some shell scripts for that.

from duplicati.

From [email protected] on April 01, 2011 11:43:55

The remote management would be something like a webservice interface where you can read/write xml like responses (or probably JSON).

I had not considered it, but of course, you can just use the OS scheduler and have it send commands like a normal cli/script and never user the internal scheduler.

The log is something I expect there to be a 100 different requests for, so I plan to make it plugable with simple python scripts, so you can easily redirect messages to your favorite destination. There will probably be stock logging for linux log, windows eventlog, file based and email.

Summary: Separate the GUI frontend, and allow Duplicati to run as a service

from duplicati.

From [email protected] on April 26, 2011 00:00:56

Status: Started

from duplicati.

From [email protected] on April 27, 2011 15:59:34

Is there any way to use existing duplicati profiles with the system scheduler on windows?

from duplicati.

From jhiswin on April 27, 2011 16:03:30

As of right now, no. After this issue is implemented it should be possible by using the cli.

from duplicati.

From [email protected] on April 28, 2011 13:02:20

If your only requirement is to replace the Duplicati scheduler with the Windows scheduler, you can create a job that runs:
"C:\Program Files\Duplicati\Duplicati.exe" --run-backup=""

It is not entirely meant to be used this way, but it will run the backup, provided that it exists. It is slightly un-stable in that it still requires a display to be attached and that there is no return code or other non-UI indication of success/failure.

If Duplicati is not running, it will start. If it is running it will activate the backup in the running instance.

from duplicati.

From [email protected] on April 28, 2011 13:23:30

That was exactly what I was after, thanks!

I'm still looking forward to the resolution of this issue, however.

And if I may put my $.02 in, I think you should punt on user management and just employ basic password-only authentication to the backend for now.

Regarding the scheduler, I would defer to the OS scheduler. That is the status quo for the *nix world, and I believe there are windows APIs for managing scheduled tasks if you wish to continue to maintain the scheduling in the UI.

Regarding communication, if I recall .NET allows you to expose methods as webservice methods fairly easily - although I'm not sure how this works in standalone applications. I would suggest this approach if you can get it "for free" with the libraries you are using. If you want to be real hip, you can structure things into a REST/JSON API, but SOAP would probably get you up and running most quickly in a .NET environment.

Anyhow, this is a great and much-needed project. Keep up the good work!

from duplicati.

From [email protected] on April 28, 2011 13:25:39

Ah didn't realize that duplicati would run continuously as a service in this model, so I suppose my OS scheduler comment is moot. Nevermind :)

from duplicati.

From [email protected] on September 20, 2011 14:23:36

Were about 2.5 years into this request. Would be nice if more effort would go into this feature than adding a bunch of crazy new storage options such as s3, skydrive, google docs, etc. I just want a stable backup/restore solution thats cross platform and can easily work in headless environments. =P

from duplicati.

From [email protected] on September 20, 2011 14:50:37

I agree this feature would be quite useful. The ability to run as a service with a separate config GUI would definitely make Duplicati a server-class application.

from duplicati.

From [email protected] on September 20, 2011 14:52:20

Re #26: It's a volunteer project. If that's what you want, your non-demanding, non-condescending help would be appreciated. I'm sure that the author would welcome solid patches.

Yeah, I want this too, but you get more flies with sugar than vinegar, yes yes.

from duplicati.

From [email protected] on September 21, 2011 00:47:18

Thanks for letting me know that this is still requested :)
(Warning: There is a lot of text following....)

I do what I can to keep this project running, but it is still, largely, a one-man-show.

If you refer to the creation date for this issue, it was back when 1.0 was just released, and that version had a lot of problems that I tried to address with 1.2. I was hoping that 1.2 would be stable enough so that I could focus on 2.0 (thus the service split), but instead I am trying to do a minor 1.3 release with some additional problems adressed. Unfortunately I do not work as fast as I had hoped, but I only work on this in my spare time, and there has to be time for family too.

You might wonder why I seem to have enough time for minor tasks, but not focus on the larger issues. The reason is that the smaller issues are much more immediate, and I can go in and make a GoogleDocs backend in a few hours. It does not matter much if I am interrupted while doing it, because it is almost mechanical work for me. An issue like issue #9 requires a lot of planning and it takes a long time where I have to keep track of all the different sub-issues. If I am interrupted while doing this, chances are that something breaks horribly, and thus I need to know that I have at least a few hours before I can work on this.

Just because there has been no progress externally on this issue does not mean that I am not working on it, but I want this issue to adress some more fundemental issues with Duplicati.

First such issue is that the WinForms has to go. When I started, it looked like the Mono project were just about to ship themed WinForms components for Mac and Linux, but they have not yet, and while it mostly works, it looks like crap. Even if I could fix the Mac trayicon issue, there would never be more than a few users because it looks so horrible (worse than Win95).

Second issue is that I would like this to open the door to buisiness users that want to do remote monitoring and control of the backup.

My idea to solve this is to go all html/css/ajax with the UI, and this involves a mini webserver that basically serves all the static html/css files and has a special ajax-like interface for controlling the database, the scheduler and the currently executing backup. I have the infrastructure for this ready, but I need to implement all the ajax call methods and the html/css UI. This model will allow me to write a tray-icon native to each platform that looks (and is) 100% system-skinned, but basically calls the webserver (i.e. practically no code to maintain). The main UI (the wizard pages) will then be shown in a native window with an embedded browser, as that works on all platforms and relieves me from having to write more than one version of the pages. The html/css is also MUCH easier to work with, and I hope that this will allow more people to contribute to the Duplicati UI, as my graphical skills are limited. It will also make it very easy to write a remote monitor tool, or make a plugin to a company's existing monitor system. And html/css/ajax seems to go well with Win8.

To allow for a local executing Duplicati (portable mode), the server/webserver can be hosted in the Duplicati process (like running the server.exe). The authentication is still not in place in my head, but I plan to use a file to check user permissions when running on the same machine, and I probably have to write multiple authentication systems for the supported platforms to get remote login support.

I think it will be cool :)

While I have been doing all this thinking and making my life hard, a programmer called Kenneth Bergeron (incidentially with the same first name as me), has developed and contributed a new GUI for Duplicati, complete with a service backend. In short, this allows you to run Duplicati as a service, albeit with a slightly different GUI look. This version is built Windows-only and takes advantage of many Windows features, such as integrated authentication, database logging, etc. The source code is in trunk, under the name Duplicati.Scheduler. No release has been made yet because he is currently testing it on some systems.

This does not mean that I will skip my idea, because I think the two things cater for different user groups, but the two versions will probably merge at some point.

As for my limited time, I do not mind other people taking over. You are of course encouraged to write me if you want to contribute some code, but there is so much more to running a project than just code. I strive to answer reported issues within 24 hours, not necesarily solve them that fast (that would be impossible), but just let people know that I take their reports seriously and also make sure that I request extra information before the setup/Duplicati gets thrown away. You can also join the google-groups/mailing-list and respond to people that ask things, which will free up more of my time too. If you want to, you can also get access to edit/create wiki pages, describing how to do stuff, debugging or commandline usage for instance. I have also created a Duplicati wikipedia page that you may want to edit, and generally "spread the word" which takes up much of my time. If you are more into graphics, I could use a cool logo or some banners that can be displayed on supporting sites. And then there is planning. As you can see the roadmap is not really updated, and you can take on the role of "release manager" and decide what bugs are important to fix and which should be left for the next version. And I am sure there are a hundred other things you can do. If I feel that such things are taken care of, I will focus on producing code, otherwise it will be a cover-all-bases approach from me.

Thank you all for using Duplicati and being interested enough to actually interact with me. (Did you really read all of that text?)

from duplicati.

From [email protected] on September 22, 2011 01:54:53

@kenneth,

Thanks for the detailed response. i am definitely interested in a more central management system for multiple systems and I am willing to pay for it, but definitely please do not make any type of AIR or webserver type setup that will run on the clients end. Some backup solutions like vembu are doing it this way and i cant stand it.

Thanks for your hard work and i do appreciate it, so as i am pretty sure you already have done, please dont take my comments as anything more than feedback.

@boldandbusted,

I dont think made a single DEMAND. I 100% understand open source projects and I work actively with many of them. What i did make was a comment and request. Some people call that feedback, which is good for projects. I am not a paying customer, so i obviously have no right to make any demands.

from duplicati.

From [email protected] on December 22, 2011 14:59:21

Hello Kenneth,

I really appreciate your great effort in creating this wonderful solution, integrating various ways of data backup. I have been using it widely for my personnel use to backup my laptop with my lab work.

My comment here is in regard to your Comment #29 where you mentioned that a developer named Kenneth Bergeron has created a service based implementation of Duplicati and might publish it to open source. Do you have any idea or did he mention to you, when it might become available? Please let me know if you know already.

Thanks and best regards.

from duplicati.

From [email protected] on December 23, 2011 01:40:50

Now that we released Duplicati 1.3 we focus on 2.0. As far as I know Kenneth has already a Duplicati service running but there is no UI yet. As soon as there is something we can show, we'll put something online.

from duplicati.

From [email protected] on December 23, 2011 02:13:28

The work that Kenneth Bergeron has done is actually published as open source.
But there are no published binaries, because Bergeron wants to work/test it some more.

Just check out the current trunk, and open "Duplicati Scheduler.sln".
You need Visual Studio installed (VS Express works fine), to open and build the project.

Last I tested, there was a .ico file missing but apart from that the project builds and runs.

And yes, I do have a no-UI version of Duplicati running which exposes a web service, and I have started work on the UI, but unless you want to do actual development, don't look at it yet :).

from duplicati.

From [email protected] on February 09, 2012 11:32:23

There is a UI; but, no installation.
It has some testing done.
I have been using it for several months.
Please download it and let me know how it works.

"Just check out the current trunk, and open "Duplicati Scheduler.sln".
You need Visual Studio installed (VS Express works fine), to open and build the project."

Cheers, Ken

from duplicati.

From [email protected] on February 25, 2013 18:22:17

So...still nothing, hmm? It's been a year since the last update and we're still waiting for a way for Duplicati to run without having to be logged in.

What's the status?

from duplicati.

From [email protected] on February 26, 2013 02:45:00

There has been quite some progress, but still not near useful status.

The server component is largely complete, and runs jobs without any UI attached.
The user interface has been through some rewrites, but is still some way from being able to add or edit jobs.
We have had a good contribution that helped produce the basic dialog needed to edit/add a backup, but it needs some work still. We've been contacted by a few people wanting to help out, and we do our best to let them.

I have had much less time than anticipated, so my contributions to the project in the last year has been fairly limited, mostly responding to requests (such as this). I have side-tracked a bit and started working on a new-and-improved storage format, that solves the issues with long chains, time-consuming full backups, de-duplication etc. I need this part for my own personal needs, so once this is done, my own priority for the project returns to the UI.

If anyone wants to work on the new UI, simply download the source, and open with either Visual Studio (express), or MonoDevelop. Set the Duplicati.GUI.TrayIcon as startup project, and click "Run".

You can now let Duplicati open the webpage that is the main user interface, and start editing the extjs files. If you want commit acces, or would like to coordinate your work, write me an email.

I have attached the mockup that René has been working on, so you can see where we are heading.
I have also attached some screenshots of where we are. The *** marks stuff that is not done.

Attachment: Main Window - Actions 2012-09-30.png Screenshot2.png Screenshot4.png

from duplicati.

From [email protected] on August 28, 2013 01:55:48

Any progress since februry regarding this?

I just discover Duplicati which is a great software to make some backup but the service/daemon is really missing @ the moment.

Thx :)

from duplicati.

From [email protected] on August 28, 2013 08:26:58

We chose to postpone the GUI work, because we found that the current
version of Duplicati has some drawbacks, and we expect the GUI work to take
quite some time. The changes that are required to address these drawbacks
has wide-reaching implications, and we decided that it would be better to
improve the base version (the CLI) before comitting to the GUI. With the
CLI you can do some basic scheduled backup runs, so it can cater for our
own personal needs, and make up for the missing service functionality.

In short: No GUI updates right now, but a completely redesigned core with a
new storage engine and a new CLI.

Regards, Kenneth

from duplicati.

From [email protected] on September 04, 2013 00:37:18

Do you have any ETA for version 2.0 release.

from duplicati.

From [email protected] on September 04, 2013 04:31:01

No ETA.

We work as fast and as much as possible, but we have paid day jobs.

Anyone interested in speeding things up is most welcome to join the team.
The GUI needs someone with skills in HTML/CSS/Javascript.
We are currently using Sencha (previously known as ExtJS) as the JS framework, but we can discuss that part.
Just drop an email directly to me or René if you want in :)

from duplicati.