alphagov / paas-cf-conduit Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
I'm using conduit to supply VCAP_SERVICES
to a bash script which does some processing. If the bash script fails conduit is swallowing the exit code.
Conduit should return the exit coder from my subcommand so that I can perform additional actions if required.
Here's a simple examples:
exit.sh
#!/bin/bash
exit 1
Normal operation:
$ ./exit.sh
$ echo $?
1
Wrapped by conduit:
$ cf7 conduit db -- ./exit.sh
OK Connecting client
...
OK Waiting for conduit app to become available
OK Fetching service infomation
OK Binding db
OK Fetching environment
OK Starting port forwarding
OK Waiting for port forwarding
OK Preparing command: ./exit.sh
$ echo $?
0
Hi,
I only need to setup postgres backups about once a year, even less often, which is infrequent enough to hit the same issues.
It would be good if there was a TROUBLESHOOTING section in the README, and ideal if there was also something cf diagnose-postgres
and cf diagnose-mysql
to find common connection issues.
OK Preparing command: pg_dump -f backup.sql
pg_dump: error: connection to database "XXX-YYY-ZZZ" failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
It would be nice if there was a hint to run a diagnose command on the end, like this:
Postgres connection failed, you can try diagnosing connection issues with:
cf conduit diagnose-postgres
Where cf conduit diagnose-postgres
would check versions match and other connectivity issues (like whether the database port appears to be open, which can be achieved without even using psql)
Cheers
Stuart
Exits cleanly and stops/deletes the __conduit__
app
conduit app left in running state.
The access_token
is expiring so the plugin is no longer authorized to clean up
Hi
I've managed to get myself in a bit of a situation and I was hoping someone here might be able to help.
While testing paas, I have created a cf postgres service called pg-svc
. I have then used the conduit plugin to connect to the instance and run psql
. During the session I have then made some changes to the db as the (I believe) temporary pg role created by the conduit plugin.
Somehow though I have managed to disconnect the conduit session but the related conduit app (conduit_bupby9jm) has not been removed, it's still there in cf apps
and on the gov.paas dashboard. The temp conduit user role is also still in the pg db.
I now can't remove the conduit app because Service broker error: pq: role "ugzehw_usjxjqmqt" cannot be dropped because some objects depend on it
. Neither can I remove the pg service because Cannot delete service instance. Service keys, bindings, and shares must first be deleted.
If I try and unbind the service with cf unbind-service __conduit_bupby9jm__ pg-svc
I get the same n unbind operation for the service binding between app __conduit_bupby9jm__ and service instance xws-db failed: Service broker error: pq: role "ugzehw_usjxjqmqt" cannot be dropped because some objects depend on it
I'm not sure what the pg objects are that depend on the temp conduit role as I don't fully recall what scripts I ran but querying the db doesn't seem to report any objects actually being owned by that role. I have tried to REASSIGN OWNED BY __conduit_bupby9jm__ TO anotherrole;
but I don't have sufficient permissions and when I set the db up, I didn't specify CREATEROLE so I'm unable to give myself permission.
I hope that makes sense to someone. I can't even delete the space because of the above issues. Shy of trying to fully delete the org, I'm out of ideas. Any help would really be appreciated - thanks.
After a while conduit starts to write the following error messages:
Authorization server did not redirect with one time code
Authorization server did not redirect with one time code
[...]
I suspect the token needs to expire for this to happen.
I connected without an app:
cf conduit <service name>
Start a conduit to a shell/repl type command:
cf conduit mything -- psql
Send ctrl+c.
psql
handles the ctrl+c as it sees fit ... in this case "clear the current input"
Get booted out of the psql session and the conduit closes.
When running a foreground command, signals should be forwarded to the child command.
Hello! I'm at @18F, and one of the authors of Service Connection Plugin. I was just going to submit the plugin to the Cloud Foundry Plugin Directory, and came across yours. It seems like the two projects have the same goal - any interest in combining forces on them?
We should detect service types (postgres/mysql) based on other heuristics (such as the connection uri string), and be more flexible about detecting and extracting credentials from the binding data.
Currently we rely on the service name itself being "postgres" or "mysql" and binding credentials being in the same specific format as our rds-broker output.
This is great for us , but there's no reason why this plugin couldn't work for services provided by a different broker with different service names and slightly different binding data.
The plugin will be less useful for the wider community
org govuk-notify
, space production
, conduit name __conduit_79086__
Timeline:
conduit created (cf conduit notify-db -- psql
) at 10:23
used normally at 10:24:59
entered another query at 10:44:46 - no response
killed manually using ctrl+c at 10:48:
rdsbroker_**********=> select * from notifications where sent_by = 'mmg' limit 5;
^[[A^[[A^[[B^[[B^C
[leohemsted:~]$ Could not send cancel request: PQcancel() -- connect() failed: Connection refused
SSL SYSCALL error: EOF detected
The connection to the server was lost. Attempting reset: Failed.
!> \q
[leohemsted:~]$
10:49 conduit still visible in cf apps
If neither the client or the server send any packets for a few minutes the server seems to hang up.
We should implement some kind of keepalive so that you can use conduit for things like long-running sql queries, or interactive sessions where you're not doing anything for a while.
Go's ssh client doesn't have a magic keepalive flag, but it does have a SendRequest()
function which looks like it can achieve the same thing:
I tried to connect to a db instance as normal this morning but I encountered this error. Screenshot included for further information. I was able to connect to this instance yesterday afternoon. I've tried logging out then logging in again and re-installing the conduit plugin. Unable to connect to mysql shell instance either.
Delete the __conduit_XXXXX__
application and service binding when the utility exits.
If necessary, change the instructions to use ^D (close STDIN) rather than ^C (send SIGTERM).
So that the application doesn't use resource quotas and the service credentials are revoked.
Hello,
I am installing Conduit for the first time but keep getting a Proxy Authentication error. I am using a Windows pc in our corporate environment. I have proxy settings set in my environment variables that allow me to use PaaS and Cloud Foundry okay however I cannot install this plugin. Is there any other alternatives for this or any ideas to help me? Thanks...
Searching CF-Community for plugin conduit...
Get https://plugins.cloudfoundry.org/list: Proxy Authentication Required
FAILED
[leohemsted:~]$ env | grep PGDATABASE
PGDATABASE=notification_api
[leohemsted:~]$ cf conduit notify-db -- psql
[ ... ]
OK Preparing command: [psql]
psql: FATAL: database "notification_api" does not exist
This repo doesn't currently have any tests. That's OK as a proof of concept.
If users find it useful and depend on it then we should write tests and have them automatically run against pull requests so that it's easier to review changes with increased confidence.
Any existing environment variables are not overridden when setting up the env for executing commands.
$ PGDATABASE=xxx cf conduit posty -- env | grep PGDATABASE
PGDATABASE=xxx
PGDATABASE=rdsbroker_8xxxxx_2xxxxx_bxxx_bxxxxxxxxx
Only the single (relevant) PGDATABASE variable in the environment.
Overwrite existing environment variables before exec.
As part of the work to integrate the https://github.com/alphagov/paas-aiven-broker with the GOV.UK PaaS, we're documenting the user-facing ways of interacting with the services it provisions. cf conduit
appears to fail to provide a tunnelled-via-an-app route to these services, instead displaying the backend host which is trivially available via cf env
.
This is a problem because it doesn't work with services which have an IP whitelist in front of them and which allow only PaaS-originated traffic to reach the service instances.
In Aiven's case, this happens for the Elasticsearch service. It's exposed as an HTTPS URI, with embedded/inline credentials. Perhaps this URI is confusing conduit?
$ cf create-service elasticsearch small-ha-6.x es
Creating service instance foo in org govuk-paas / space demo as admin...
OK
Create in progress. Use 'cf services' or 'cf service es' to check operation status.
$ until cf services | grep ^es | grep -v 'create in progress'; do sleep 5 ; done ; cf conduit es
OK Connecting client
OK Targeting org govuk-paas
OK Targeting space demo
OK Deploying __conduit_8664__
OK Uploading __conduit_8664__ bits
OK Starting __conduit_8664__
OK Waiting for conduit app to become available
OK Fetching service infomation
OK Binding es
OK Fetching environment
OK Starting port forwarding
OK Waiting for port forwarding
The following services are ready for you to connect to:
* service: es (elasticsearch)
hostname: redacted-1840a56-a302-a57eeb2d3090.aivencloud.com
password: redacted
port: 10264
uri: https://786b50b1-20a0-4d06-b50b-96ca048e4b89:[email protected]:10264
username: 786b50b1-20a0-4d06-b50b-96ca048e4b89
Press Ctrl+C to shutdown.
Just want to note down one of the causes of conduit not closing down.
In some circumstances the conduit
proxy app might remain running forever.
For instance, if the cf conduit
fails, or if the cf token expires or the user targets a different API endpoint or org or space.
If the application is running forever, it might happen that the user is billied for ever.
One solution can run some kind of watchdog command in the app, that would fail if no SSH tunnel is created or if it stops. For example, dropping a safe-terminate.sh
#!/bin/bash
ssh_running() {
netstat -t | grep -q ":2222"
}
start_timeout=$(date -d "+1 min" +%s)
max_timeout=$(date -d "+7 days" +%s)
while ! ssh_running && [ "$(date +%s)" -lt "${start_timeout}" ] ; do
echo "Waiting for SSH session..."
sleep 5
done
while ssh_running && [ "$(date +%s)" -lt "${max_timeout}" ] ; do
echo "SSH session still running..."
sleep 600
done
echo "No more SSH sessions or timeout detected. Exitting in 10 seconds."
sleep 10
then, cf conduit
would push the static app as follows:
cf push -b staticfile_buildpack -m 64m -k 64m -i 1 --health-check-type none --no-route --no-manifest __conduit_12345__ -c 'bash $HOME/public/safe-terminate.sh'
The app would wait for a SSH to be present in 1 minute, and then wait until it is gone. Then it will terminate after 10 seconds. In normal operation, the flow would be as usual, the application would be deleted once conduit finished.
If instead the cf conduit
fails to delete the app, the SSH tunnel would eventually die. The script would detect that and terminate. CF would restart the application several times, but the script would fail after one minute because no SSH connection is done.
Publish to https://plugins.cloudfoundry.org/ when we're happy user feedback and quality.
Hi,
I am developing an application on the GOV.UK PaaS for BEIS.
I have a GOV.UK PaaS account, and I have created a database with the following commands:
cf login -a api.cloud.service.gov.uk -u *** -p ***
cf target -o ***
cf create-space int
cf target -o *** -s "int"
cf create-service mysql Free myapp-database
# wait until the database service is ready
cf install-plugin conduit
When I attempt to connect to the database with the following command:
cf conduit myapp-database -- mysql
I get the following error:
ERROR 1045 (28000): Access denied for user 'un73dmsmf7wo7anm'@'10.0.34.10' (using password: YES)
Does anyone know how I can diagnose or fix this issue?
Many thanks,
Rich
Conduit allows running a command. For instance if connecting to a Postgres database you can directly run psql
against it by cf conduit your-postgres -- psql
. But Conduit does not warn you if running psql
(or the equivalent for other services) against the wrong type of service. This has confused PaaS support and is reasonably likely to confuse other people.
When downloading the latest version of cf:
cf install-plugin conduit
I'm receiving version 0.0.15 instead of the expected version 0.1.0
When running one of the example scripts:
cf conduit <INSTANCE> -- pgdump -f backup.sql
I receive this error:
Error: Unknown program pgdump: can't determine what service types it expects
This happens for docker also when using the docker example.
A team mate that had this previously working (on version 0.0.12) is also experiencing the same issue after upgrading.
Temporary solution found by my team mate:
cf install-plugin https://github.com/alphagov/paas-cf-conduit/releases/download/v0.0.14/cf-conduit.darwin.amd64 -f
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.