New value for retention policy to define a minimum number of archives to keep. When scanning for archives to purge, we should count the most recent good archives and delay the purge if the minimum number is not met.

Use case: Site that "hibernates" part of the year, backups may be turned off so new backups would not be created and after the retention period all archives would be purged meaning zero backups would exist. With a minimum of "3" set for example, the last three would be retained until new backups started again. This would allow the site's DB to be removed while hibernating and the last "good" archive to be restored when the site came out of hibernation.

Use case: Failing backups. With short retention dates, if the backups fail repeatedly for a that length, all backups would be purged. Failed backups should be resolved as quick as possible but when that is not possible, it would be beneficial to keep the last few "good" backups for emergencies.

Since we're no longer listing UUIDs anywhere except the raw JSON, it probably makes more sense to include the names of jobs/etc. in the logs. So instead of:

2016-01-30 00:01:01.660993032 +0000 UTC shieldd: INFO: initial run of dbb41a2b-614b-4332-ab97-553aa1c3b71f (daily at 9:00) is at 2016-01-30 09:00:00 +0000 UTC

We might have:

2016-01-30 00:01:01.660993032 +0000 UTC shieldd: INFO: initial run of 'Cloud Foundry Backup' (dbb41a2b-614b-4332-ab97-553aa1c3b71f) (daily at 9:00) is at 2016-01-30 09:00:00 +0000 UTC

Implement a mechanism for cleaning up in-progress tasks on the agent side. In theory, if the worker closes the ssh connection to the agent, the agent will receive some sort of messaging, and can then attempt to clean up + cancel the shield-pipe exec. Propagate some sort of signal (sigpipe?) from the agent through shield-pipe to the plugins.

Update the plugin framework to perform a cleanup function (added to the plugin interface), when the signal is received.
Update all the plugins to implement the cleanup function

Verify that this results in a canceled tasks from the cli actually canceling processes on the agent (requires #31 to be complete)

This may be related to #55, and may have been fixed. However, as of v2.1, there is an issue somewhere where backup jobs can fail on the agent, report the output back to the daemon, but somewhere along the line, the task gets reported as a success in the database, and the archive is marked as valid. Investigate why this is, and fix, providing necessary tests.

Currently Shield does not do anything with tasks that never come out of the pending state. To address this:

Add a function to ReschedulePendingTasks:

• Find all tasks in the database listed as pending.
• Cross-reference those tasks with the tasks in the scheduling queue of the supervisor. If there is a database-found task which is not in the scheduling queue, add it to the queue

Call this function at startup

Thanks for sharing this great work with the community!

We've been working on a very similar backup/restore system targeted at CF users, and exposed as a service broker. See specs and current early implementation at https://github.com/Orange-OpenSource/service-db-dumper

/cc @ArthurHlt

Would a service broker UI something you'd be interested in, as a complement to Shield Web and CLI UIs ?

We started looking into how to have the service-db-dumper leverage shield as a backend, with the following expected benefits:

• enable support for additional backends type as they added to shield
• provide sophisticated schedule & retention policies to service-db-dumper users:
  • schedule and retention policy would be passed as an additional arbitrary param into the "create-service-instance"
  • Note the retention policy support would likely need to be completed with store size quotas (i.e. max size GB for all dumps in addition to an expiration date. Reaching the max size could either fail the dump, or automatically delete older dumps).

We started imagining such an integration as:

• when service-db-dumper starts it would use the store API to setup an S3 store when dumps would be uploaded/downloaded
• when the service-db-dumper receives a provision request to perform a dump it would:
  • create a new target using the target API to dump from
  • request an adhoc backup (not sure how ?): by
    • register an immediate one-off schedule through the schedule api ??
    • register a single backup retention policy
    • schedule a job with immediate schedule and retention policy
    • poll the job completion
• when the service-db-dumper receives a update request to restore a dump it would:
  • optionally a new target using the target API, to restored to
  • request a restoration through the archive api
  • find the job associated th the scheduled restoration using the task api
  • poll the job completion

Would you confirm this would be the right way to design such an integration ?

Looking into the shield code and tests and the current POC phase, it seemed too early to start prototyping such an integration, but we're interested in starting the conversation with the shield team, both at the product spec level and software design level.

Thanks!

Guillaume.

Now that we've bundled a psql binary in the form of pgtools with shield-agent, have the postgres plugin use that, so users do not have to worry about the symlink correct pathing to find the psql binary on the server being backed up, when creating target configs

If there are pending tasks in shield list tasks --all, they show up with a Started field of (pending), and a Stopped field of (running). Seems impossible for un-started jobs to be running, we probably need to change this wording to (not yet started) under the Stopped column, but only for pending tasks.

$ shield list tasks --all | head
UUID                                   Owner      Type     Remote IP           Status     Started                           Stopped
====                                   =====      ====     =========           ======     =======                           =======
106bf8a7-84d1-4673-8b53-fe933300e6db   gfranks@   backup   10.244.3.46:5444    pending    (pending)                         (running)

Every X ticks (configurable?), have the supervisor kick off the purge routine, to do the following:

1. Sweep the db for archives with state 'valid' that have lived past their retention period indicates they should live for, and mark their state as 'expired'
2. Sweep the db for all archives with state != 'purged' and state != 'valid', and execute a purge on them
   If the purge is successful, mark the archive state as 'purged', and set the purge-reason to the previous value of the archive state (likely either 'expired', or 'invalid')

Additionally, restrict the default response of the archive listing api to only show those of state 'valid', but allow querying to get invalid/purged archive data

Lastly, update the workers to mark archives with state 'invalid' if the worker task failed (either failure during backup or storage).

Write a shield plugin to support dockerized-logsearch service broker backups. Should back up the elasticsearch data for each elasticsearch container/service, as well as the service broker data (containers, credentials, port assignments). Restore should be a seemless restore + go.

Should accept and emit JSON blobs for the list / show / create / edit / etc. actions. These are to be used by people attempting to automate their SHIELD installation. No niceties for the robots.

→  ./shield ls archives

Shield API IP:Port is unknown, specify the API endpoint using one of:

    export SHIELD_API="127.0.0.1:8080"; shield ls
    SHIELD_API="127.0.0.1:8080" shield ls
    shield -H "127.0.0.1:8080 ls"
→  SHIELD_API="10.244.2.2:8080" ./shield ls archives
Get 10.244.2.2:8080/v1/archives?status=valid: unsupported protocol scheme ""
→  SHIELD_API="http://10.244.2.2:8080" ./shield ls archives
UUID   Target   Remote IP   Store   Taken at   Expires at   Status   Notes
====   ======   =========   =====   ========   ==========   ======   =====

If a protocol needs to be specified, then the help dialogue should probably convey that, as not to imply that http is assumed by default.

Add a new channel for workers to listen on called the suicide channel.
Add the suicide channel to the task struct
When canceling a task, issue a specific message to the channel, to indicate that the task should terminate. Log that the task is being killed
Add a goroutine for workers to listen to on the suicide channel of the task. if the correct message is received, close the ssh client connection to the agent. Emit a log message indicating what is happening

Ensure that this breaks the worker out of any blocks it has while executing that task, and that it moves on to process the next task.

Right now all archives are listed regardless of their number, so when there are a large number of archives (say for a large number of services with nightly backups) running shield ls archives will just list all the archives even if the total # is in the thousands. Recommend only outputting the first X with an option to --all. and/or adding an option to let the user specify how many to list, e.g. shield ls archives --recent 20.

We have code like this all over:

var params struct {
  /* some structure def */
}
json.NewDecoder(req.Body).Decode(&params)

// use `params`, without checking for bad JSON formatting, non-JSON input, etc.

We should handle errors from the JSON decoder, as 400 Bad Requests.

Current if you just run shield it only said unrecognized command

$ shield
unrecognized command

Right now it is not hooked up. Follow standard practices for debugging:

• Implement a DEBUG(string, ...interface{}) function
• Print to os.Stderr
• Prefix each line with DEBUG>

At the very least, we should DEBUG() that we have enabled debugging output.

Update the shield CLI to be a little more friendly:

• When listing archives, include the archive state (valid/invalid/expired/etc)
• When listing anything with target data (targets, tasks, jobs, archives), include the agent IP of the target
• When running a restore option on the implicit target, have the client retrieve that info from the DB, and publish it (along with agent IP) vs restoring to ''
• When displaying agent data, have the CLI refer to it as the 'Remote Agent' or 'remote_agent', rather than 'SHIELD Agent', or 'agent'

TASK FAILED!! shield worker 3 failed to execute the command against the remote agent 10.244.2.2:5444 (Process exited with: 16777216. Reason was: ())

Processes don't exit with codes that high. Fix the parsing of the error message.

When creating retention policies, targets, etc. the new policy/target/etc. doesn't create with the error "unexpected end of JSON input".

$ shield create policy
Policy name:
test
Policy summary:
test
Policy expiration in seconds (protip: there are 86400 sec per day):
1000000
ERROR: Could not create new retention policy: unexpected end of JSON input

When restoring archives, the shield CLI attempts to filter the archives presented to the user based on the job selected. This filtering appears to be slightly broken:

$ shield restore archive
More than one job matched your search for '':

      Name              Summary                      Target            Store   Schedule
      ====              =======                      ======            =====   ========
   1) redis-dedicated   dedicated-vm redis backups   redis-dedicated   s3      hourly at :22
   2) redis-shared      Shared-vm redis backups      redis-shared      s3      hourly at :22


  Which backup job do you want? [1-2] 2


Here are the 4 most recent backup archives for redis-shared:

      UUID                                   Taken at                          Expires at                        Status   Notes
      ====                                   ========                          ==========                        ======   =====
   1) d73925a8-2bf2-4af9-a5b0-3c80cc4d3544   Tue, 02 Feb 2016 22:32:55 +0000   Wed, 03 Feb 2016 22:32:55 +0000   valid
   2) 97f5ddf4-73aa-4604-af3a-7200699b396e   Tue, 02 Feb 2016 22:32:43 +0000   Wed, 03 Feb 2016 22:32:43 +0000   valid
   3) 8106a4ab-6606-4826-b14b-3e1426c2fd28   Tue, 02 Feb 2016 22:22:03 +0000   Wed, 03 Feb 2016 22:22:03 +0000   valid
   4) f53db303-5767-4eb9-876d-482683e0c10b   Tue, 02 Feb 2016 22:22:03 +0000   Wed, 03 Feb 2016 22:22:03 +0000   valid


  Which backup archive would you like to restore? [1-4] 2


Scheduled immediate restore of archive '97f5ddf4-73aa-4604-af3a-7200699b396e'

This shows 4 backups for redis-shared. However 2 are for redis-shared and 2 are for redis-dedicated:

$ shield list archives
UUID                                   Target                           Remote IP          Store     Taken at                          Expires at                        Status   Notes
====                                   ======                           =========          =====     ========                          ==========                        ======   =====
d73925a8-2bf2-4af9-a5b0-3c80cc4d3544   redis-dedicated (redis-broker)   10.244.3.54:5444   s3 (s3)   Tue, 02 Feb 2016 22:32:55 +0000   Wed, 03 Feb 2016 22:32:55 +0000   valid
97f5ddf4-73aa-4604-af3a-7200699b396e   redis-shared (redis-broker)      10.244.3.46:5444   s3 (s3)   Tue, 02 Feb 2016 22:32:43 +0000   Wed, 03 Feb 2016 22:32:43 +0000   valid
8106a4ab-6606-4826-b14b-3e1426c2fd28   redis-shared (redis-broker)      10.244.3.46:5444   s3 (s3)   Tue, 02 Feb 2016 22:22:03 +0000   Wed, 03 Feb 2016 22:22:03 +0000   valid
f53db303-5767-4eb9-876d-482683e0c10b   redis-dedicated (redis-broker)   10.244.3.54:5444   s3 (s3)   Tue, 02 Feb 2016 22:22:03 +0000   Wed, 03 Feb 2016 22:22:03 +0000   valid

I'm seeing this in my logs:

2016-01-29 11:51:44.046665315 -0500 EST ./shieldd: ERROR: shield timed out task '2d937045-5633-47f0-bf3f-dcf097cfd9b6' after running for 12h0m0s
2016-01-29 11:51:44.047123435 -0500 EST ./shieldd: ERROR: shield timed out task 'f7bae52e-f9ee-4b2c-a97e-2e5d7bb6870c' after running for 12h0m0s
2016-01-29 11:51:44.047639214 -0500 EST ./shieldd: ERROR: shield timed out task '3147a608-3a80-4de2-b23a-a448530704fb' after running for 12h0m0s
2016-01-29 11:51:44.048105094 -0500 EST ./shieldd: ERROR: shield timed out task 'ff29dd32-61f9-4fbd-8052-dac1da484bd4' after running for 12h0m0s

The same block repeats every second, filling up disk.

Investigate and fix.

When shield starts up, check for any records in the tasks table that are of state 'running'. For each found:

• Fail the task
• If the task is a 'backup' operation, and has an associated archive/restore key, trigger a purge for that archive.

When setting the environmental variable SHIELD_TARGET if there is a trailing / then POSTs and PUTs will fail, although GETs continue to function normally. When a user creates the environmental variable:

export SHIELD_TARGET=http://a.b.c.d:pppp/

Should be read as http://a.b.c.d:pppp.

Add a plugin for taking mysql backup/restores (specifically for the purpose of backing up an RDS instance), but generic if possible.

Provide documentation along-side each shield plugin, indicating what type of plugin it is, what it's designed to be used with, what the json configuration should be, and details of how each action is performed.

Been watching with anticipation. How about an introductory post on Gopher Academy's much read December Blog Post series? https://docs.google.com/spreadsheets/d/1H0rj7TJo0t8GWyNdGwEc6bGcDj6UgJUN42JtAtbPMrk/edit?usp=sharing

It should exit 1, and print a noisy message to stderr with the following info:

• This is SHIELD
• No, that's not a real sub-command
• Here are some real sub-commands
• Here is how you use the help

Provide a shield plugin to backup/restore/store/retrieve/purge elasticsearch snapshots to/from s3. Should backup elasticsearch data directly, as if it's a stand-alone (non-service-broker-based) system.

The owner of a restore task gets set to 'anon'. Ad-hoc backup tasks are set to the user running the backup. We should propagate this in for ad-hoc restore tasks as well.

UUID Owner Type Remote IP Status Started Stopped ==== ===== ==== ========= ====== ======= ======= 85f93041-5c56-4eaa-8577-2e43940e0d48 anon restore done Mon, 25 Jan 2016 14:44:11 +0000 Mon, 25 Jan 2016 14:44:16 +0000 67c2d46c-0adb-411a-9af3-9864f3c414f4 gfranks@ backup 10.72.72.21:5444 done Mon, 25 Jan 2016 14:37:50 +0000 Mon, 25 Jan 2016 14:37:52 +0000

e.g.

$ shield ls archives
UUID                                   Target                              Remote IP          Store     Taken at                          Expires at                        Status   Notes
====                                   ======                              =========          =====     ========                          ==========                        ======   =====
e17a4df0-7bf5-43c3-ac69-450ddc96f595   docker-postgres (docker-postgres)   x.y.72.61:5444   s3 (s3)   Fri, 22 Jan 2016 09:00:50 +0000   Sat, 21 Jan 2017 09:00:50 +0000   valid
15b55b49-ddf4-4cbf-8a3c-b75cc4a47620   docker-postgres (docker-postgres)   x.y.72.61:5444   s3 (s3)   Thu, 21 Jan 2016 17:38:06 +0000   Fri, 20 Jan 2017 17:38:06 +0000   valid
17e2b6b8-e427-4726-a2eb-fe916fbf64b8   docker-postgres (docker-postgres)   x.y.72.61:5444   s3 (s3)   Thu, 21 Jan 2016 17:01:16 +0000   Fri, 20 Jan 2017 17:01:16 +0000   valid

$ shield edit target
Target Name (docker-postgres):
Summary (sb docker-postgresql93 target):
Plugin Name (docker-postgres):
Configuration ({}):
Remote IP:port (x.y.72.61:5444): x.y.72.125:5444


Target Name:    docker-postgres
Summary:        sb docker-postgresql93 target
Plugin Name:    docker-postgres
Configuration:  {}
Remote IP:port: x.y.72.125:5444


Save these changes? [y/n] y

Updated target
Name:          docker-postgres
Summary:       sb docker-postgresql93 target

Plugin:        docker-postgres
Configuration: {}
Remote IP:     x.y.72.125:5444


$ shield ls archives
UUID                                   Target                              Remote IP           Store     Taken at                          Expires at                        Status   Notes
====                                   ======                              =========           =====     ========                          ==========                        ======   =====
e17a4df0-7bf5-43c3-ac69-450ddc96f595   docker-postgres (docker-postgres)   x.y.72.125:5444   s3 (s3)   Fri, 22 Jan 2016 09:00:50 +0000   Sat, 21 Jan 2017 09:00:50 +0000   valid
15b55b49-ddf4-4cbf-8a3c-b75cc4a47620   docker-postgres (docker-postgres)   x.y.72.125:5444   s3 (s3)   Thu, 21 Jan 2016 17:38:06 +0000   Fri, 20 Jan 2017 17:38:06 +0000   valid
17e2b6b8-e427-4726-a2eb-fe916fbf64b8   docker-postgres (docker-postgres)   x.y.72.125:5444   s3 (s3)   Thu, 21 Jan 2016 17:01:16 +0000   Fri, 20 Jan 2017 17:01:16 +0000   valid

Add support to shield to time-out tasks and mark them as failed if they take too long.

If a job gets scheduled but has something currently marked as running, fail that running job, with a message about a timeout occurring, prior to starting the new job. If possible, retain any of the stdout/stderr of the job that is being timed out, and append on the timeout message, storing it all in the task log. Use the 'failed' state. Ensure that a task timeout error also shows up in the shield log, in case anything is parsing that log for failure messages + monitoring

When I created a job, I accidentally set yes for paused. I tried to edit job, to make it unpause, the confirmation info says pause is false, I type y to apply the change, however when I list jobs, it is still set to "Y" to pause.

Now that create policy accepts input in days and shows the retention schedule in days, need to clean up how the time is shown:

$ shield create policy
Policy Name: one-year
Summary: Retain for one year
Retention Timeframe, in days: 365


Policy Name:                  one-year
Summary:                      Retain for one year
Retention Timeframe, in days: 31536000


Really create this retention policy? [y/n] y

Created new retention policy
Name:       one-year
Summary:    Retain for one year
Expiration: 365 days

The main target plugins like PostgreSQL used a STDOUT fed into the STDIN of a source plugin like S3

This works when the backup can dump to STDOUT like "pg_dumpall" or "tar"

Solutions that are integrated, like elasticsearch, should to have a way to specify that the plugin handles both parts and can accept the target and store (and require both). This could call the single plugin and skip the STDOUT->STDIN piping.

Multi-line strings should be wrapped on newlines in multi-columnar tabular output, so that we don't wrap at the edge of the screen, implicitly.

Add support for running backups every hour. Should be able to specify them (case insensitively) via:

For running on the 15th minute:
hourly at 15
hourly at :15
hourly at *:15
hourly at h:15
hourly at x:15
every hour at 15
every hour at :15
every hour at *:15
every hour at h:15
every hour at x:15

1. If the user did not target shield, can we out put something like set your target" shield -H ${shield_ip:port} [options] (command} instead of simply saying Get https://shield/v1/targets: dial tcp: lookup shield: no such host.

2. If the user did set the target, but it is a wrong ip, then we will out put something like "check your shield target ip exist".

3. when we type `shield", the help list should include set SHIELD_TARGET part.

4. Rename SHIELD_TARGET to SHIELD_API (avoid confusion which target plugin and shield_target_endpoint)

5. Formatting shield output to be more human friendly

e.g.

$ shield rm policy single
More than one retention policy matched your search query of 'single':

      Name                   Summary   Expires in
      ====                   =======   ==========
   1) Default Retention                100 days
   2) Single Day Retention             1 days


  Which retention policy do you want to use for this backup job? [1-2] 2


Name:       Single Day Retention
Summary:
Expiration: 1 days

It would be nice if I could run shield status and get back some confirmation that I am talking to a real shieldd endpoint.

If the connection fails, the sub-command should exit non-zero and print a message about the problem

If it succeeds, some basic (non-privileged) information about the shield should be returned, à la bosh status:

SHIELD v{whatever}
{endpoint}

OK

Implement a new /v1/status endpoint that provides this information.

Right now it is not hooked up. Follow standard practices for debugging:

• Implement a DEBUG(string, ...interface{}) function
• Print to os.Stderr
• Prefix each line with DEBUG>

At the very least, we should DEBUG() that we have enabled debugging output.

If two fields of a target are inadvertently switched:

$ shield show target 67c24213-31b7-495d-a6b5-d2131e5ffc03
UUID:         67c24213-31b7-495d-a6b5-d2131e5ffc03
Name:         sandbox-bosh
Summary:      Sandbox BOSH Database

Plugin:       postgres
Endpoint:     a.b.c.d:pppp
SHIELD Agent: {"pg_user": "vcap","pg_password": "c1oudc0w","pg_host": "localhost","pg_port": "5524","pg_bindir": "/var/vcap/packages/postgres-9.4.2/bin","pg_dump_args": ""}

Then the target cannot be edited or deleted. The shield CLI output will indicate that the edit or deletion is successful, but using show target or ls targets shows the target still present and unedited.

This issue may also apply to stores as well, for a similar reason.

When creating a new job, update the default value for 'paused' to be 'no', rather than 'yes'.

when backing up redis instances that are active, the plugin does not handle the backup gracefully. tar errors when the appendonly.aof file is updated during backup, or if services are deleted.

This should be addressed.

Sometimes, the API component of the supervisor makes immaterial changes (i.e. renames, annotation) to the database that Supervisor doesn't need to resync on.

Update the API layer to only poke the supervisor for important data changes.

In the shielddb:

shielddb=# SELECT * FROM archives;
                 uuid                 |             target_uuid              |              store_uuid              | store_key |  taken_at  | expires_at | notes | purge_reason
 | status
--------------------------------------+--------------------------------------+--------------------------------------+-----------+------------+------------+-------+-------------
-+--------
 17e2b6b8-e427-4726-a2eb-fe916fbf64b8 | 15993168-0419-4811-82b6-9a5959bbb508 | d9402c2a-7b0d-4325-b680-174c7b5f08f4 |           | 1453395676 | 1484931676 |       |
 | valid
 15b55b49-ddf4-4cbf-8a3c-b75cc4a47620 | 15993168-0419-4811-82b6-9a5959bbb508 | d9402c2a-7b0d-4325-b680-174c7b5f08f4 |           | 1453397886 | 1484933886 |       |
 | valid
(2 rows)

Currently when attempting to restore the archive this does not result in an error, e.g.:

$ shield restore archive 17e2b6b8-e427-4726-a2eb-fe916fbf64b8
Scheduled immedate restore of archive '17e2b6b8-e427-4726-a2eb-fe916fbf64b8'

• Archives created with empty store_key's should be marked as invalid
• Attempting to restore an invalid archive should error, client side
• Restrict listing of archives to valid archives only, unless --invalid is specified (new flag)
• New command: purge invalid archives to delete everything that is invalid

1. It would be nice to alias shield restore to shield restore archive, similar to what we do with list policies and list retention policies.
2. If no archive uuid is provided in a restore command, query the API and get a list of valid archives for restore, for the end user to choose.

Build a concourse pipeline for running the unit tests, managing versions and creating GH releases of (at the very least) the SHIELD CLI.

This may also need to feed a pipeline for the BOSH release.