tlux / sftp_client Goto Github PK
View Code? Open in Web Editor NEWAn Elixir SFTP Client that wraps Erlang's ssh and ssh_sftp.
License: MIT License
An Elixir SFTP Client that wraps Erlang's ssh and ssh_sftp.
License: MIT License
Found it !
That happens when I make and run a prod release (Elixir 1.9.0):
set MIX_ENV=prod mix release
then myprogram start
then when I try a command (myprogram rpc "mycommand(args)":
=> crash and
** (UndefinedFunctionError) function SFTPClient.connect/1 is undefined (module SFTPClient is not available)
When I had :sftp_client to applications or extra_applications options in mix.exs, everything is OK.
Did I forget something when I made the release ?
That doesn't happen with iex -S mix (and yes, the config files have the same content: dev.exs == releases.exs ;) )
When trying to install in Livebook desktop on Mac with the following command:
Mix.install(
[
{:sftp_client, "~> 1.4"}
],
force: true
)
The following error is received:
== Compilation error in file lib/sftp_client/ssh_xfer_attr.ex ==
** (RuntimeError) error parsing file /Applications/Livebook.app/Contents/Resources/rel/vendor/otp/lib/ssh-5.0.1/src/ssh_xfer.hrl, got: {:error, :enoent}
(elixir 1.15.7) lib/record/extractor.ex:84: Record.Extractor.read_file/2
(elixir 1.15.7) lib/record/extractor.ex:50: Record.Extractor.extract_record/2
lib/sftp_client/ssh_xfer_attr.ex:8: (module)
could not compile dependency :sftp_client, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile sftp_client --force", update it with "mix deps.update sftp_client" or clean it with "mix deps.clean sftp_client"
In diving deeper on this, it is because the vendor OTP that is installed with Livebook desktop does not include the src
folder for ssh-5.0.1
. Is this something that can be changed in a future version?
First, thanks a lot for SFTPClient !
I'm currently developping a SFTP Client/Server and after using another SFTP library, I give a try to yours at the moment.
I made some tests on a local Windows PC (2 SFTP servers on differents ports exchanging files).
The transfer rates are all very nice (from 23 to 41 Mb/s), streamed or not...
Except for direct upload (SFTPClient.upload_file function) for which the rate falls down to 0,6 to 0.9 Mb/s only.
I isolated yours functions (SFTPClient.connect and SFTPClient.upload_file) in case this fall could come from my own code but with the same results.
Do you have an idea about what could cause that ?
Best regards.
For the traveler who is struggling to get their private key to work:
you need to generate the key-pair in PEM format. For instance, if you using ssh-keygen
to create your keys: ssh-keygen -t rsa -m PEM -f id_rsa -C"[email protected]"
ed25519 does not seem to be compatible, at this time, with this client and/or the underlying ssh_sftp client. rsa, however, works.
I found that when I specified a user_dir
param with client config, ssh_file would generate a known_hosts file in the user_dir path with an algorithm I didn't even specify. Not sure what's going on with that! So, I am not using this parameter when connecting:
{:ok, conn} =
SFTPClient.connect(
host: "localhost",
port: 2022,
user: "username",
private_key_path: "absolute/path/to/private_key",
sftp_vsn: 2
)
Good luck.
For Elixir noobs (like me ;) ):
Mentionning to also add :
def application do
[
applications: [:sftp_client, ...]
...
]
end
in the readme.md ?
Hi! First of all, thanks for the great work!
I bumped into an exception in a non-bang function running a simple download pipeline, when using code like
case SftpClient.stream_file(conn, path) do
{:ok, data} -> do_something_with(data)
{:error, reason} -> {:error, reason}
end
** (SFTPClient.OperationError) Operation failed: permission_denied
(sftp_client) lib/sftp_client/operation_util.ex:37: SFTPClient.OperationUtil.may_bang!/1
(elixir) lib/stream.ex:1392: anonymous fn/5 in Stream.resource/3
(elixir) lib/enum.ex:3023: Enum.reverse/1
(elixir) lib/enum.ex:2668: Enum.to_list/1
(myapp) lib/worker.ex:433: Worker.download/2
Can't access the sftp at the moment, but the error is pretty clear
I'm using 1.3.5 but didn't see any related diff in last commits
Didn't dig into the problem yet, pretty noob in elixir, but will try to take a look (and in the meanwhile add some defensive try rescue :P )
Hi.
I'm using OTP 23 and cannot connect to the server. The problem is here:
warning: :ssh_file.add_host_key/3 is undefined or private. Did you mean one of:
* add_host_key/4
lib/sftp_client/key_provider.ex:12: SFTPClient.KeyProvider.add_host_key/3
warning: :ssh_file.is_host_key/4 is undefined or private. Did you mean one of:
* is_host_key/5
lib/sftp_client/key_provider.ex:15: SFTPClient.KeyProvider.is_host_key/4
The dirty fix by adding an additional parameter to both functions works for me, but I'm not sure how to fix it properly to preserve the compatibility with older OTP versions.
Error to connect to SFTP
The following arguments were given to :ssh.connect/4:
# 1
'host'
# 2
"9003"
# 3
[
user_interaction: false,
user: 'admin',
silently_accept_hosts: true,
quiet_mode: true,
password: 'admin!',
key_cb: {SFTPClient.KeyProvider,
[private_key_path: nil, private_key_pass_phrase: nil]},
inet: :inet,
connect_timeout: 5000
]
# 4
5000
call:
{:ok, conn} = SFTPClient.connect(host: "host", user: "admin", password: "admin", port: "9003")
Hi. Thanks for the library!
While updating an application to Elixir 1.16.1 I got a dialyzer error on an sftp_client funcion call. This made me try running dialyzer on this repo (with the same Elixir version) and I got the following
lib/sftp_client.ex:207:no_return
Function upload_file!/3 has no local return.
________________________________________________________________________________
lib/sftp_client/operations/upload_file.ex:31:no_return
Function upload_file!/3 has no local return.
________________________________________________________________________________
lib/sftp_client/operations/upload_file.ex:32:call
The function call will not succeed.
File.stream!(_local_path :: any(), [], 32768)
breaks the contract
(Path.t(), :line | pos_integer(), [stream_mode()]) :: File.Stream.t()
________________________________________________________________________________
done (warnings were emitted)
Halting VM with exit status 2
This happens because on Elixir 1.16.0 the File.stream!(file, options, line_or_bytes)
was deprecated and the spec reflects only the latest version. The issue elixir-lang/elixir#13212 has more details and a possible solution.
After this removal of a deprecated function ( which was only there for debugging apparently ):
erlang/otp@b39f33e
the following crashes when reading a file:
14:48:47.726 [error] Process #PID<0.269.0> raised an exception
** (FunctionClauseError) no function clause matching in :file_io_server.file_request/2
(kernel 10.0) file_io_server.erl:227: :file_io_server.file_request(:pid2name, {:state, {:file_descriptor, :prim_file, %{handle: #Reference<0.1131244897.1132068896.124240>, owner: #PID<0.269.0>, r_buffer: #Reference<0.1131244897.1132068865.127900>, r_ahead_size: 0}}, #PID<0.268.0>, #Reference<0.1131244897.1131937793.127901>, "", :binary, :latin1})
(kernel 10.0) file_io_server.erl:185: :file_io_server.server_loop/1
:file_io_server.file_request(:pid2name, ...) does not exist anymore.
this will show up in a stacktrace as follows:
stacktrace:
(ssh 5.2) ssh_sftp.erl:1043: :ssh_sftp.read_file/3
(sftp_client 2.0.1) lib/sftp_client/operations/read_file.ex:17: SFTPClient.Operations.ReadFile.read_file/2
As the behavior of stream_file/1
may be confusing (see #8), the function should be deprecated. Instead stream_file!/1
should be used.
@doc """
Reads a file from the server, and returns the data as String.
"""
@spec write_file(Conn.t(), Path.t(), String.t() | [String.t()]) ::
:ok | {:error, SFTPClient.error()}
def write_file(%Conn{} = conn, path, data) when is_binary(data) do
write_file(conn, path, [data])
end
@doc
:ok
or {:error, _}
according to @spec
data
is_binary according to implementationdata
can be String.t or [String.t] according to @spec
Hi there,
I’m experiencing something weird. I can connect to SFTP servers just fine, all the operations I’ve tried work, but when I do a list_dir command, it hangs forever if the folder contains no files.
For example this:
{:ok, conn} = SFTPClient.connect(get_in(state, [:config, :server]))
{:ok, stat} = SFTPClient.file_info(conn, "in/images/products")
IO.puts("Stat result #{inspect(stat)}")
{:ok, list} = SFTPClient.list_dir(conn, "in/images/products")
IO.inspect(list)
SFTPClient.disconnect(conn)
If the "in/images/products"
folder contains files, everything is hunky-dory, I get a list back.
However, if "in/images/products"
has no files (either being completely empty, or having just subfolders), list_dir does not return. The following lines are not executed, I’ve tried waiting at least 10 minutes, much longer than my SFTP timeout, so something’s weird.
Let me know if there’s anything I can do to help reproduce this problem.
Hi there.
Is there any way to pass an additional option to the underlying :ssh_sftp.start_channel/3
call? I have to connect to a pretty outdated server and for that reason I need to modify configuration algorithms via the modify_algorithms
option. It seems to be not possible right now since the option has to be listed in the Config struct.
HI,
I'm trying to make a SFTP connection with private rsa key.
The service works and i can reach it with CLI and filezilla.
I tried a lot of config combination. The key is a classic id_rsa
. the keys are generated with ssh-keygen
command on cli.
I tried with something like [host: 'sftp', user: 'user', user_dir: 'keys', private_key_path: 'keys/id_rsa']
and also implementing manually a key_cb
but i receive the same error everytime.
The error i get is this one:
10:16:06.539 [info] Erlang SSH :client 4.9 (OpenSSL 1.1.1d 10 Sep 2019).
Server: 'SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u4'
Disconnects with code = 14 [RFC4253 11.1]: Unable to connect using the available authentication methods
State = {userauth,client}
Module = ssh_connection_handler, Line = 931.
Details:
User auth failed for: "user"
{:error,
%SFTPClient.ConnError{
message: "Unable to connect using the available authentication methods"
}}
Looking at the State
in the error, it seems that is missing the public/private key authentication method, but i can't find a solution.
Do you have some ideas why this doesn't work? What am I doing wrong?
Thanks
Hi i'm trying to upload a file using SFTPClient.upload_file!(conn, local_path, remote_path)
.
but i'm getting
(Protocol.UndefinedError) protocol Collectable not implemented for %SFTPClient.Stream{chunk_size: 32768, conn: %SFTPClient.Conn{channel_pid: #PID<0.661.0>, config: %SFTPClient.Config{connect_timeout: 5000, dsa_pass_phrase: nil, ecdsa_pass_phrase: nil, host: "host_ip", inet: :inet, key_cb: {SFTPClient.KeyProvider, [private_key_path: nil, private_key_pass_phrase: nil]}, operation_timeout: :infinity, password: nil, port: port, private_key_pass_phrase: nil, private_key_path: nil, rsa_pass_phrase: 'secret, sftp_vsn: nil, system_dir: nil, user: 'user', user_dir: 'keys_dir'}, conn_ref: #PID<0.659.0>}, path: "remote_path"} of type SFTPClient.Stream (a struct). This protocol is implemented for the following type(s): Ecto.Adapters.SQL.Stream, Postgrex.Stream, IO.Stream, HashDict, File.Stream, Map, MapSet, HashSet, BitString, List (elixir 1.10.2) lib/collectable.ex:1: Collectable.impl_for!/1 (elixir 1.10.2) lib/collectable.ex:77: Collectable.into/1 (elixir 1.10.2) lib/stream.ex:531: Stream.do_into/5 (elixir 1.10.2) lib/stream.ex:649: Stream.run/1 (sftp_client 1.4.0) lib/sftp_client/operations/upload_file.ex:37: SFTPClient.Operations.UploadFile.upload_file!/3
what do i am doing wrong?
This is not a bug, but adding this for anyone experiencing the same issue in the future.
OTP 24 disabled public key ssh-rsa per default: erlang/otp@59285df and it needs to be added manually if you are using it.
Is it worth it to add update the readme to add ssh-rsa to the config when upgrading to OTP 24?`
modify_algorithms: [
append: [
public_key: [:"ssh-rsa"]
]
]
what is right way to define ip interface for outgoing connections?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.