Converting to Github Actions or Azure Pipelines. about sqlalchemy-exasol HOT 13 CLOSED

This would be a major step forward for automating the build. Builds are currently ran against an EXASOL cluster hosted by Exasol. We have a test user there but need to be careful in how we expose the credentials (only to core committers). Also, this limits parallelism in the build and we cannot manage the EXASOL cluster version as this is done by EXASOL.
So running this on GitHub Actions that allows spinning up an own EXASOL docker image sounds quite attractive and would allow a lot more contributors to run builds.

from sqlalchemy-exasol.

Yes, I'm hoping to find some time to work on this soon.
The work I have done is on this branch: https://github.com/vamega/sqlalchemy_exasol/tree/github-actions

I ran into an issue while trying to debug why the Exasol docker container wasn't starting up on Github Actions. I'm suspecting it might be something related to the docker containers requirement of O_DIRECT which may not be supported on the runners that Github provides.

I made a community post about it here: https://github.community/t5/GitHub-Actions/Debugging-service-container-startup-failure/m-p/33908, but never got a response.

I haven't found the time recently to dig into this, and I think it's going to be at least two weeks before I'll have the time to dig into it. I'm not particularly familiar with Github Actions which makes this something I'm slower at doing.

However it's something I want to see done, so I'm glad you're open to the switch. Makes me more motivated for finding the time to finish it.

from sqlalchemy-exasol.

Looks promising. Also make sure to reach out to the folks at EXASOL, maybe they have an idea on why the EXASOL container does not spin up.

from sqlalchemy-exasol.

@vamega have you made any progress? I was trying to read up on how services in GitHub actions are configured. I stumbled on this post that refers to setting the workdir options: --workdir /home/dynamodblocal
https://github.community/t5/GitHub-Actions/Github-Actions-services-not-reachable/td-p/30050
However, I am not sure if EXASOL is sensitive to a home directory (have not found anything in their documentation).

from sqlalchemy-exasol.

@BY-jk - Sorry my own work has been very busy and I haven't had the chance to investigate this more thoroughly.

I think an option like workdir might help, since I suspect the issue is the need for O_DIRECT support for the filesystem used to store the Exasol data.

I was hoping to replicate that by using a mergerfs partition at home which I know does not support O_DIRECT when the parameter direct_io is set to false.

I have seen your post on the Exasol docker-db repository, hopefully they will be able to provide more insight into this.

My question on the Github community site hasn't received any traction. What I'd really like is the ability to debug the docker container that is run in the pipeline, but I don't know of a way to do that at the moment.

from sqlalchemy-exasol.

@vamega if you are still on this. EXASOL commented on this in their docker image project over here: https://github.com/exasol/docker-db/issues/44#issuecomment-616532017

from sqlalchemy-exasol.

from sqlalchemy-exasol.

More context was provided, especially a hint at an exasol docker image to take as a starting point.
https://github.com/exasol/docker-db/issues/44#issuecomment-617083378

from sqlalchemy-exasol.

Thanks for pointing that out.
Will take a stab at this over the weekend.

from sqlalchemy-exasol.

@BY-jk , only managed to do some reading on this over the weekend.
The repository linked may not be appropriate for what Github Actions, but I think I now know how I can configure the base docker container to test my hypothesis.

Will keep this issue updated with any progress I make.

from sqlalchemy-exasol.

@BY-jk @vamega I am the maintainer of the mentioned repository. Github Actions doesn't provide O_DIRECT which can be disabled in the configuration. We can try this and then use it for integration testing.

from sqlalchemy-exasol.

@BY-jk I opened a draft pull request for Github Actions with Exasol Docker-DB. It is currently, unfinished and I might need some input from you to finish it.
It uses integration-test-docker-environment, because it greatly simplifies the start of the configured docker-db's and also waits for the database to be actually available. It also supports the configuration for different Exasol versions which is usually a bit cumbersome, because the config file format changed with the Exasol versions. We are using it also in our own test system for the Script Language Container, as such it will be maintained also in the future.

from sqlalchemy-exasol.

close with #91

from sqlalchemy-exasol.