This gem allows you to setup regular extractions from a SFTP server. It's expected that the server you're connecting to has file structure like shown below and that your user has privileges to copy and delete files from it's home directory:
<USER'S DEFAULT DIR>
`-- some_folder
|-- PENDING
| |-- 2014_09_24_customers.csv
| `-- 2014_09_25_customers.csv
`-- PROCESSED
|-- ...
|-- 2014_09_20_customers.csv
|-- 2014_09_21_customers.csv
|-- 2014_09_22_customers.csv
`-- 2014_09_23_customers.csv
It's suggested you use the commands as shown here to setup your crontab or whenever gem.
Add this line to your application's Gemfile:
gem 'sftp_extractor'To test the sftp-extractor you'll need to have a SFTP server running your machine. To do that on Mac OS X follow this instructions.
Where it says 'Allow access for:' by default is Administrators which I changed for my username.
Test your port 22 is now opened:
$ nc -zv localhost 22
# => Connection to localhost port 22 [tcp/ssh] succeeded!or simply,
$ sftp [email protected]
Password:
Connected to 127.0.0.1.
sftp> dir
Applications Desktop Documents Downloads Google Drive Library Movies Music Pictures Publicsource /srv/rails-env/env.sh && cd /srv/sftp-extractor && time ruby bin/sftp_extractor.rb -c conf/campaign.yml -e $rails_envruby bin/sftp_extractor.rb -c conf/template.yml -e development| key | description | optional | |
|---|---|---|---|
| credentials | :server | SFTP server hostname/IP | |
| :user | username | ||
| :pwd | password | ||
| :timeout | parameter accepted by Net::SSH constructor (applied only during the CONNECT) | ||
| root | relative path to the default user's home, where are stored the files you want to extract | ||
| folder | :in | list of patterns to filter the files you want to extract | |
| :local_out | relative path where the extracted files are should be saved | ||
| :on_success[:move_to] | after successfully extracting the files they'll be moved into this folder on the SFTP server | X | |
| :on_success[:cleanup] | period of time after which files are going to be deleted from folder[:on_success][:move_to]. Example: 6 hours |
X | |
| :extraction_mode | Options: - last: extracts the most recent file matching the PATTERN; - full: extracts every file matching the PATTERN |
X | |
| retry_on_error | :period | if at least one of the files wasn't successfully extracted, waits this amount of time | X |
| :max_times | no. of times is going to wait period time to retry the missing patterns |
X |
# conf/campaign.yml
production:
credentials:
server: xx.xxx.xxx.xxx
user: xXxXxXx
pwd: xXxXxXx
timeout: 20
root: "."
folder:
in: { root: PENDING, patterns: ["*customers.csv"] }
local_out: incoming/campaign
on_success:
move_to: PROCESSED
cleanup: 6 hours
extraction_mode: full
logger:
path: /var/log/sftp-extractor/campaign.log
level: info
retry_on_error:
mail_config:
to:
- [email protected]
- [email protected]
subject: "Extraction from SFTP failed (CUSTOMERS LIST)"
body: "Execution Log:"- Fork it
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Add some feature') - Push to the branch (
git push origin my-new-feature) - Create a new Pull Request
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent