lightninglabs / faraday Goto Github PK

The granularity of price data from coincap is limited by the period of time you're querying; more time/less granular data is allowed. Right now, api calls to get prices from coincap require specifying the granularity and guessing until you have the right granularity for your period. This should just happen in the background.

Steps to Completion

1. Create max-granularity which maps a period of time to a maximum granularity level (considering that we allow splitting up of queries into 5 parts)
2. Remove granularity from the rpc and just automatically set it.

Step 1 of this task is also relevant to accounting report generation, since longer periods may just fail to get price data with our current default value (eg, if you ask for a report of 1 year).

Running the frcli audit command fails somehow arbitrarily. For some start/end times it works fine, for some it doesn't and returns a 401 error.

I can audit reports for any time of the year (months, quarter of, half of and full year).

Running frcli audit --loop-category --start_time 1672527600 --end_time 1680299999 for the first quarter of 2023 works just fine. But frcli audit --loop-category --start_time 1680300000 --end_time 1688162399 for the second quarter fails with the error:
[frcli] rpc error: code = Unknown desc = status code: 401, response: ""

Running it without --loop-category has the same behavior.

But running just frcli audit works just fine.

See above. We are running faraday in Kubernetes.

Standalone.

CLI

We build our own image, currently we are using 0.2.11-alpha

Runs on a kubernetes cluster.

Runs on a kubernetes cluster.

I can't find a faraday.log file, where do I get it?

It would be great if I could supply raw certificates and macaroons to Faraday, instead of paths to said certificates and macaroons. I know that this is handled by lndclient library, so I filed an issue for this there, with more motivation: lightninglabs/lndclient#29

Also filing this issue to get it on your issue tracker, if the issue in lndclient is fixed.

When you use openchannel with the push_sats argument (ie. open channel worth 10 BTC and push 5 BTC to the other side from the beginning state)

I doubt anyone will use push_sats on mainnet in prod... but in our test environment it was very obvious since we use push_sats all the time, and everything is wonky because of it.

Also, should we be considering LOCAL_CHANNEL_OPEN capacity as debited funds?

Currently it shows amount: 10 BTC, credit: false for the channel open when I still control half the funds in the channel.

Perhaps LOCAL_CHANNEL_OPEN should only "debit":

1. Any amount pushed in first state via push_sats
2. Any amount that is not usable on our end (set aside for closing fees etc.) (These "set aside" things can be credited during close if we don't use them all)

frcli version like lncli version would be really nice.

lightningnetwork/lnd@f93a8ab

$ faraday --version
unknown flag version' 2020-06-27 14:11:32.019 [INF] GVNR: Error starting faraday: error loading config: unknown flag version'
$ frcli --version
frcli version 0.2.0-alpha commit=v0.2.0-alpha-31-g81e9e530e8c1361d428baab83cbf2f4aa9da7f66
$ frcli version
No help topic for 'version'

As is, the fiat package only uses CoinCap's API for price information. The package is designed to allow different price sources without affecting our price calculation logic.

Steps to completion

1. Identify different sources of price data and propose in this issue (perhaps coindesk)
2. Add code to obtain price data over a range and update fiat/GetPrices to allow users to choose a backend
3. Update rpc to allow choice of backends

Expected behavior

The ability to send a request with a CUSTOM backend and custom_prices via REST.
Docs here say REST position unknown

Actual behavior

I can't figure out a proper way to send the request.

To reproduce

For example: (change values accordingly)

curl \
    -k \
    -g \
    -H 'Grpc-Metadata-macaroon:'$(cat ~/.faraday/regtest/faraday.macaroon | xxd -ps -c 99999999) \
    https://localhost:8466/v1/faraday/nodeaudit?start_time=1674089890\&fiat_backend=CUSTOM\
\&custom_prices[0][price]=4000000\
\&custom_prices[0][price_timestamp]=1674089870\
\&custom_prices[0][currency]=JPY\
\&custom_prices[1][price]=4100000\
\&custom_prices[1][price_timestamp]=1674089930\
\&custom_prices[1][currency]=JPY\
 && echo ""

Changing custom_prices to something like custom_prices=glorp it gives me a parse error for BtcPrices... so it's expecting it via the query parameters.

The solution might be to somehow allow this data to be included in the body of the request as JSON or something...

Even if we somehow got this working, 108 characters per entry will blow up and hit some URL query limits eventually...

During the recent call with several lnd users, the notion of being able to obtain a balance "snapshot" at any period of time came up. I think the use case here is going back and looking at the historical flow of funds within a channel.

FWIW, this is already available in lnd as is, since we store the entire "revocation" chain. However this isn't indexed by "time" and instead uses an auto-incrementing sequence number of each new channel state. The only notion of time for the revocation chain is the absolute expiation heights on any HTLCs possibly present on the commitment transaction.

Alternatively, this can be done by polling lnd periodically, or having faraday use the streaming balance RPCs to update its own local state.

Following on from #23 transaction reporting.

The parent issue for this task produces reports for our on-chain behaviour, categorizing on chain transactions as channel opens, closes, receipts or payments. The last two categories combine user initiated sends/receives with on chain resolutions (sweeping of anchors, claiming of htlcs etc).

Further split this list out by identifying our own sweep transactions.

$ frcli nodereport
[frcli] rpc error: code = Unknown desc = payment htlc has a route with zero hops

My node is quite old and has some early payments with no hops stored.

I had a similar problem with joule that was fixed.

joule-labs/joule-extension#207

Versions of faraday and lnd:

frcli version 0.2.0-alpha commit=v0.2.0-alpha-9-gdf39c5a5c9e27d6935d14db1f1c0e7ad2197acc4
lnd version 0.10.99-beta commit=clock/v1.0.0-66-ged93c16e51abfd3512cb9a2800811b598f6768cd

Moving this here: lightningnetwork/lnd#1253

make && make install should install all dependences, but I get the below error.

Fully installed libraries and dependencies.

go: git.schwanenlied.me/yawning/[email protected]: git fetch -f git://git.schwanenlied.me/yawning/bsaes refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /home/umbrel/go/pkg/mod/cache/vcs/7eea7e6da5639dca452594d1b27f1f302ecc350d3818bf7255e110b8cdc8d151: exit status 128:
	fatal: unable to connect to git.schwanenlied.me:
	git.schwanenlied.me[0: 2a03:b0c0:2:d0::1013:d001]: errno=Permission denied
	git.schwanenlied.me[1: 188.166.73.239]: errno=No route to host
go: error loading module requirements
GO111MODULE=on go build -v -ldflags "-X github.com/lightninglabs/faraday.Commit=v0.2.6-alpha-25-gf4dd97a0fddea558e7f2da8b17f8e698c9335d64" github.com/lightninglabs/faraday/cmd/faraday
# cd .; git ls-remote https://git.schwanenlied.me/yawning/bsaes
fatal: unable to access 'https://git.schwanenlied.me/yawning/bsaes/': SSL: certificate subject name (sekigahara.schwanenlied.me) does not match target host name 'git.schwanenlied.me'
# cd .; git ls-remote git+ssh://git.schwanenlied.me/yawning/bsaes
[email protected]: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
# cd .; git ls-remote ssh://git.schwanenlied.me/yawning/bsaes
[email protected]: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
go: finding git.schwanenlied.me/yawning/bsaes.git v0.0.0-20180720073208-c0276d75487e
go: git.schwanenlied.me/yawning/[email protected]: git fetch -f git://git.schwanenlied.me/yawning/bsaes refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /home/umbrel/go/pkg/mod/cache/vcs/7eea7e6da5639dca452594d1b27f1f302ecc350d3818bf7255e110b8cdc8d151: exit status 128:
	fatal: unable to connect to git.schwanenlied.me:
	git.schwanenlied.me[0: 2a03:b0c0:2:d0::1013:d001]: errno=Permission denied
	git.schwanenlied.me[1: 188.166.73.239]: errno=No route to host
go: error loading module requirements
make: *** [Makefile:72: build] Error 1```

<!--- How reliably can you reproduce the issue, what are the steps to do so? -->
Tried twice, still happening. 

<!-- Are you using Faraday as a standalone application or as part of Lightning Terminal? -->
On Umbrel, but installing it through command line.

<!-- If you are using Faraday as part of Lightning Terminal, did you install that using Umbrel? -->

through command line. 
<!-- How are you interacting with Faraday? Through the command line interface or the Lightning Terminal UI? -->
command line. 
<!-- What version of Faraday are you using, where did you get it (website, self-compiled, etc)? -->
pulled from git repo. 
<!-- What type of machine are you observing the error on (OS/CPU and disk type)? -->
Raspberry Pi running Umbrel. 
<!-- What is your operating system and its version? -->
Latest Umbrel. 
<!-- Any extra information that might be useful in the debugging process. -->
<!--- This is normally the contents of a `faraday.log` file. Raw text or a link to a pastebin type site are preferred. -->

Can the faraday functionality be made available over APIs (REST preferably), to enable integration with UI solutions like RTL?

Create reports for on-chain and off-chain activity of a node, similar to those produced by ln-accounting. For every off and on chain transaction that occurs within the period specified, generate an accounting entry. Transactions that pay fees will have a separate entry added for the fees paid.

Entry Format

• Timestamp: unix timestamp
• OnChain: whether the transaction was on or off chain
• Amount: expressed in msat for the sake of a consistent unit across on/off chain
• Type: type of transaction, see below
• TxID: transaction id on chain,
• FromID: source of funds, when available
• ToID: destination of funds, when available
• Fiat: fiat equivalent
• ID: unique identifier, if available
• Note: additional note to provide context

Entry Types

Each transaction added to the report should have a type which describes to the reader what lightning related operation the transaction was a part of. The first iteration of these reports will have the following types:

• ChannelOpen: channel opening transaction
• ChannelClose: channel closing transaction
• HtlcForward: fees earned from forwarding off chain
• Receipt: receipt on chain, invoice settle off chain
• Payment: balance spend from wallet on chain, settled payment off chain
• CircularRebalance: special case of off chain send, where destination is our own node, only record fees
• ChannelOpenFee: fees paid to open a channel
• ChannelCloseFee: fees paid to close a channel
• RoutingFee: fees paid off chain for our own payments
• ChainFee: fees pain on chain for our own payments

This classification falls short for on chain transactions, because we cannot distinguish between sends the user initiated on chain and sweeps produced by lnd on chain. This distinction is out of scope for this task. Follow up tasks will work on splitting these categories out further.

Export

These reports should be queryable via rpc, or cli interface. The ability to export directly as a CSV should also be supported, as this is a common use case for this kind of report.

Example output:

Expected behavior

Using the same tls.cert path as lncli commands should not error

Setting --lnd.tlscertpath={path to lnd cert} produces "no such file or directory"

To reproduce

OS X 11.4
Build LND from source
Build Faraday from source

Start Faraday w/
./faraday \ --lnd.macaroonpath={full path to lnd's readonly.macaroon} \ --lnd.tlscertpath={path to lnd cert} \ --lnd.rpcserver={host:port of lnd's rpcserver}

moving the tls.cert to ~/Library/Application Support/Lnd and using --lnd.tlscertpath="" resolves the issue

The on chain transactions required to fully resolve a channel on chain will be persisted by lightningnetwork/lnd/4157. However, this data will only be present for channels that are closed after the node upgrades to a version of lnd running this change. This leaves many projects without tools to accurately account for previously closed channels.

Given a channel ID, examine the closing transaction (and subsequent transactions) on chain to determine the resolutions that occurred on chain. This will require a connection to a node with --txindex enabled.

• Optionally allow faraday to connect directly to a node ()
• Add ChannelResolutions endpoint which optimistically queries lnd
• If no resolutions are persisted, and Faraday is connected to a node, get resolutions from chain
• For first iteration, keep resolutions in memory; consider persisting further on

Why was RPC over HTTP chosen in proto files?

I ask because I'm not able to connect to faraday with generated client. I tried C# and also JavaScript. And I tried same way as it is described here: https://github.com/lightningnetwork/lnd/blob/master/docs/grpc/javascript.md but of course with proto for faraday.

Exception is thrown on any RPC method call. E.g.

client.ChannelInsights({}, new Metadata(), {}, function (response) {
	console.log(response)
})

For JS the error is:

For C# the error is:

And it is not connectivity issue because from client (my PC) I can see opened port 8465:

Or if I'm using proto files incorrectly then please show me how to generate client in languages such as JS or C#. Thank you.

This line panics if no timestamps are given to ExchangeRate:

IMO it would be better to return an error here.

Edit:

I had a slight brain fart, meant the exchange rate endpoint and not the node audit endpoint.

This is more of a question, than an issue.

It will be super convenient if Faraday is an integrated solution with LND.
Running multiple servers is challenge for building an integrated UI solution. With Loopd and now Faraday, the list of servers required to run lightning node operations is ever growing.

Or, if your plan is to enable integration with other lightning implementations like Eclair or C-Lightning, then it may make sense to keep it as a separate server.

We run into a ResourceExhausted error while running the NodeAudit report.

Error received from peer ipv4:10.9.82.14:8465","file":"/var/local/git/grpc/src/core/lib/surface/call.cc","file_line":1075,"grpc_message":"grpc: received message larger than max (214449486 vs. 209715200)","grpc_status":8}

• The error does not seem to be dependant on the requested time range.
• We run the report with DisableFiat:true and UnknownGranularity.
• This error occurs on only one of our nodes presumably the one with the most onchain transactions.
  • We do use our LND node to send and receive onchain transactions
  • Looking through the report code, it seems that all transactions are requested from the LND Server in a single RPC call which could be over the maxMsgRecvSize of 200MB and would explain it occurs no matter what the requested time range.

Expected behavior

Node Audit should produce the report.

Actual behavior

Node Audit should produce the report

To reproduce

This error now occurs systematically on one node when we run Node Audit

System information

• Connecting through gRPC
• Version: Faraday v0.2.11-alpha

At present, the node report output query provides BTC values quoted in millisatoshis.
Multiple amount fields can be confusing, so rather than add more fields with our new units, we can deprecate the existing field in ReportEntry and replace it with a oneof. Since fiat values are subject to rounding (if we round down to BTC, we likely also want to round down our USD amount), we should paid the amount + fiat fields. Calculation in the actual accounting package should remain in msat, since it provides us with the best level of precision.

Steps to completion

1. Add a unit enum to NodeReportRequest which defaults to msat but allows satoshis and bitcoin, round accordingly in rpcReportResponse and make sure the headings in the csv file are updated
2. Deprecate the amount and fiat fields and replace with a oneof:

entryAmount{
  amount = 1;
  fiat = 2;
}

oneOf{
  entryAmount msatAmount =1;
  entryAmount satoshiAmount = 2; 
  entryAmount bitcoinAmount =3; 
}

This change should be mindful of the intention to add other fiat currencies, as detailed in #38

A user struggled to identify a transaction on https://mempool.space using the Short Channel Id provided in the CSV file. Consider also providing the txid.

(This is just a placeholder issue and may well be rewritten later on this week.)

If the price API interface is simple enough, we'd like to implement it and use our exchange's data to price the data in the audits.

1. Should be able to select currency other than USD (ie. JPY)
2. Should be able to point the price endpoint to our own API (internal, that supplies historical JPY price data)

frcli audit doesn't work due to "invalid" invoices with missing preimage. From what I see this happens when you send AMP invoices (I used them to rebalance my own channels sending to myself). Other features work as expected.

Expected behavior

frcli audit returns something meaningful.

Actual behavior

[frcli] rpc error: code = Unknown desc = ListInvoices failed: invalid preimage length of 0, want 32

To reproduce

• Send an AMP invoice to yourself using lncli sendpayment --amp
• run frcli audit

System information

Standalone Faraday, compiled myself from master. Running on Debian 10.9 amd64.

Additional information

I used lncli listinvoices --max_invoices 100000 | jq '.invoices|map(select(.is_amp == true and .r_preimage == null))' to see if my theory is correct and indeed, there are many invoices with both is_amp set and r_preimage being null. Filtering by .is_amp != true and .r_preimage == null and .is_amp == true and .r_preimage != null yields no results.

Currently users are required to specify a directory for LND's macaroons with --lnd.macaroondir. This can be problematic or annoying if you aren't running Faraday on the same server that's running LND. I would suggest the ability to use a single macaroon (like admin.macaroon) to make it easy to use. Eventually once the macaroon baking changes land in LND we could even bake a macaroon specific for Faraday.

I run c-otto.de which has around 240 channels with various characteristics. I've been running faraday for a while now (more than a year?) and never ever saw any outlier, aside from the "uptime" metrics. This cannot be right, which is why I'm raising this issue. I wasn't able to attract any attention in Slack.

Expected behavior

frcli outliers --revenue or frcli outliers --volume sometimes show some outliers.

Actual behavior

No recommendations are shown.

To reproduce

• Start the node, start faraday, wait a bit (7 days)
• frcli outliers --revenue --outlier_mult 1.5 --min_monitored 604800 | grep true -C 1000
• frcli outliers --volume --outlier_mult 1.5 --min_monitored 604800 | grep true -C 1000

System information

Standalone, CLI access. I'm running v0.2.5-alpha from GitHub.
Linux 5.10.0-8-amd64, 4 vCPUs, 16 GByte RAM, SSD.
Debian Stable.

I did:

go get -d github.com/lightninglabs/faraday
cd $GOPATH/src/github.com/lightninglabs/faraday
git clone
make && make install

Set up Faraday to connect to bitcoind backend. Seems to work.

I run faraday. Then in terminal2 I do frcli insights. Memory + Swap fills up and faraday crashes. Frcli command tells me then [frcli] rpc error: code = Unavailable desc = transport is closing.

I have 96 active channels. Server specs:

# free -m
              total        used        free      shared  buff/cache   available
Mem:           1919         522        1135           1         261        1294
Swap:          1639         511        1128

@guggero suggested to open an issue ;)

It appears the the default amount of seconds for a channel to be monitored before it gets considered by the outliers subcommand is 2419200, or 28 days.

It also appears that if the LND daemon is restarted, that this monitored time is reset back to zero seconds.

I've already had a few kernel updates this month that required system reboots, and the monitored time ended up resetting as expected.

I guess it would require the operator to monitor the values somewhat frequently (which is okay, this is a lightning node), but values such as uptime ratio still get reset to 0.

It's possible some operators might never make it to 28 days of uptime either without needing a reboot or service restart of some sort, and they may not run into the --min_monitored command flag.

I set my minimum uptime to ~2-3 days but I am hoping to get insight from other users or devs on a suggested shorter time than 28 days.

Thank you

Onchain transactions that only pay to addresses controlled by the node are not reported. From the point of view of accounting this seems correct. However, the paid onchain fees are not reported either. These fees affect the onchain balance of the node and thus should be included in the audit report.

Zero-amount onchain transactions are explicitly excluded here:

I agree that any sweep transaction of the remote counterparty should not be included since it was the remote counterparty that paid the fees. However, that condition is also excluding the transactions I described above.

As is, we only obtain fiat quotes in USD. More options would be nice for non-dollar denominated users.

[Possible solutions, still in progress]
Investigate whether coincap (currently used price backends) provides other pairs. If not, look at other backends (also relevant for #37) or look at converting BTC -> USD -> Other Currency. Preferably the first option, since each conversion loses accuracy.

The CLI uses "audit" to provide access to the "NodeAudit" endpoint. I propose to use the same name for both.

Related: #92

The newly added HtlcEventStream added to lnd's routerrpc provides new insight into a node's forwarding activity. This data is useful for a variety of applications - rate limiting for forwards, success rate tracking of various channels, rebalancing hints which trigger when we see a drop in incoming traffic or a rise in ErrInsufficientBandwidth failures.

This issue proposes the addition of a rounting package that consumes forwarding events, and matches them to their settle/fails where applicable. Such a solution could also write htlcs to disk, but for now we will start in memory and let requirements take form. A rough version of this change has been started here.

Steps to Completion

1. Update lndclient to consume forwarding events (in loop repo)
2. Consume htlc events, classifying them into forwards, receives and link failures (where forwards contain a forwarding event and subsequent settle/fail events.
3. Add the ability to make requests to this subsystem for a set of events over a period of time, which are handled by the main event loop consuming htlc events from lnd so we don't need any locking.

Is your feature request related to a problem? Please describe.
For using the Docker Image, it would be way easier to be able to just pull the image from dockerhub (or any alternative).

Describe the solution you'd like
A Docker Image is automatically created (maybe through GitHub Actions?) and pushed to a openly available registry. The Documentation is adjusted accordingly.

Describe alternatives you've considered
Let all users build the Docker images themselves.

Additional context
Especially useful for a kubernetes setup!

The current accounting report splits transactions into lightning-relevant categories (channels, circular payments etc).
This reporting system would be more extensible if we allowed creation of custom categories based on user-set label:category combinations.

For example, if a node performs loop-out swaps, they could label their on-chain sweeps with "swap" and run the accounting report with "swap":"lightning-loop" to produce a new reporting category which places all the labelled on chain swap sweeps lightning-loop category.

• The label of an on chain transaction
• The memo for an invoice
• Off chain payments don't currently have labeling functions, would likely need lnd changes

This label could also be a regex, to allow more complex matching.

Copying across from #67 made by @githorray.

Updated to the latest commit. Getting a new error:

$ frcli --version
frcli version 0.2.0-alpha commit=v0.2.0-alpha-90-g2415b5d866838880f7ddc0f57a79e3c9e589dbda
$ frcli nodereport
[frcli] open : no such file or directory

Running on mainnet.

Currently, only command args are supported, and the command gets super long.

Please support conf file parsing from default ~/.faraday/faraday.conf file.

The Harmony Reporting Standard repo has been removed by Picks & Shovels.

Linked to from this repo here and from Lightning Labs' docs here.

I found @alexbosworth 's fork here but it hasn't been updated since 2018 so there might be a more up-to-date copy out there somewhere.

I'm integrating Faraday into an application that communicates with gRPC, and I'm running into issues when importing the generated Faraday Go code: panic: proto: file "rpc.proto" is already registered. See this related issue on the golang/protobuf tracker: golang/protobuf#1122.

This is the line that panics:
https://github.com/lightninglabs/faraday/blob/master/frdrpc/rpc.pb.go#L1630

I'm no expert on technicalities of Protobuf/gRPC, but I see that other Go sources I'm using that's compiled from Proto files don't have the problematic reigstration function. These Go sources are compiled with a newer gRPC stack, specifically protoc-gen-go 1.25.0 and protoc 3.13.0. Could it be that compiling the Faraday Go sources with the newer gRPC stack would solve this?

The only real workaround right now is setting an environment variable, and I would have to make sure that this environment variable is properly set on all developer machines, production machines and CI systems. Sounds like a hassle...

In #152, the lnd version that faraday uses was bumped from v0.15.-0-beta to v0.15.4-beta. This then caused some of the faraday itests to break. Specifically:

1. The closeChannel method started to use a zero value fee. This has temporarily been solved by ignoring the used fee and instead using a hardcoded fee value. See 1 and 2
2. The second issue is that force closes no longer show up as confirmed in the itest. This could possibly due to the first issue? Perhaps the zero fee txs dont get accepted into the mempool?

Is your feature request related to a problem? Please describe.
I am currently testing my lightning setup on plebnet playground network, which is a great training grounds where payments are automated to simulate a busy network, and which is based on signet... but "signet" is not among the allowed values for the --network parameter, so I am not able to have a working installation of faraday

Describe the solution you'd like
adding the possibility to run faraday on signet, to test automation scripts based on channels profitability

Describe alternatives you've considered
selfmade scripts accessing stats from lncli or grpc, just not as good as faraday by far

Additional context
nothing to add, really :)