Convert Schwab csv to import to GhostFolio? about export-to-ghostfolio HOT 70 CLOSED

No problem, and whenever you get a chance to look at the attached, it
Schwab Test.csv
would be great. No pressure.

from export-to-ghostfolio.

That looks more like it! I'll notify you once I have something you can take a look at!

from export-to-ghostfolio.

WAHOOOO!
NO Errors!

YAY! Now I will play with the more extensive set of data.

VERY cool and MANY thanks Dick!

from export-to-ghostfolio.

Ohhhh - the numbers look valid!

Now to test, but WOW! Thank you so much for your quick turn-arounds!

from export-to-ghostfolio.

If you can provide an example export of Schwab with the likes of buy/sell/dividend orders, I can take a look at it!

from export-to-ghostfolio.

from export-to-ghostfolio.

from export-to-ghostfolio.

For some reason I don't see the attachement you sent me. Can you try again?

from export-to-ghostfolio.

from export-to-ghostfolio.

Hello-

I am not sure if you received the attachment (not showing here, but it was in the email). Please let me know if you'd rather I upload it directly to github.

from export-to-ghostfolio.

I just checked again but there isn't an attachment added to the email I received from GitHub.

I think the easiest wat would be if you upload it here.

I have some commitments this week but thereafter I will look into adding the converter.

from export-to-ghostfolio.

Closed by accident (fat finger problem)

from export-to-ghostfolio.

@DCPDad I just downloaded the file to check the contents. Are these transactions or positions? Looks like the latter. To process an export, the file should contain buy/sell/dividend transactions. These can then be converted to Ghostfolio format and be imported.

from export-to-ghostfolio.

Good question! They should be Transactions. BUT I have created another dummy CSV (attached) that might be clearer
Schwab sample2.csv
?
I really appreciate your efforts on this.

from export-to-ghostfolio.

Hi @DCPDad!

I have created an exporter for Schwab. You can find it here. Would you be so kind to test it and give me feedback as to how it performs?

I have made some assumptions as to how records should be interpreted. As I'm European I'm not familiar with the the different dividend options there were present in the sample you provided. To summarize: I have interpreted reinvest shares as a BUY order (for Ghostfolio), and any other transaction (like Qual Div Reinvest Dividend, Long Term Cap Gain) I have categorized as a DIVIDEND for Ghostfolio. Please correct me if I have misinterpreted something so I can fix that!

If all is well, can you provide me with step-by-step instructions how to make a Schwab export (like here), so I can add it to the readme? Thanks!

from export-to-ghostfolio.

from export-to-ghostfolio.

I don’t see errors attached, so I don’t know exactly what is going wrong for you. To verify: have you created an .env file with the file path pointed to your export?

As for your other points, I’ll add those to the instructions!

from export-to-ghostfolio.

from export-to-ghostfolio.

from export-to-ghostfolio.

Great!

That fee error should be an easy fix, I'll try to find some time tomorrow.

from export-to-ghostfolio.

from export-to-ghostfolio.

Hello Dick!
Just checking in to see if you had a chance to look at the 'fee error'?
Thank you!

from export-to-ghostfolio.

Should be fixed now! Can you try again?

from export-to-ghostfolio.

@DCPDad Did you have a chance to check if the tool works for you?

from export-to-ghostfolio.

@DCPDad Did you have a chance to check if the tool works for you?

I am hoping to test it this weekend!
Thank you for creating this.

from export-to-ghostfolio.

Thanks, please let me know if you find anything!

For your information, I saw some issues with dividend records with Schwab, so I just fixed those. I also added support for fee records when importing to Ghostfolio. Could you verify if the resulting Advisor Fee records are imported correclty?

from export-to-ghostfolio.

Thanks, please let me know if you find anything!

For your information, I saw some issues with dividend records with Schwab, so I just fixed those. I also added support for fee records when importing to Ghostfolio. Could you verify if the resulting Advisor Fee records are imported correclty?

Thanks Dick.

I tried but didn;t get too far. I am sure it is "pilot error" as I am not very familiar with using VS Code. I received this error after typing npm run start schwab

[email protected] start
nodemon schwab

[nodemon] 3.0.2
[nodemon] to restart at any time, enter rs
[nodemon] watching path(s): src/**/*
[nodemon] watching extensions: ts,js
[nodemon] starting ts-node ./src/index.ts schwab
sh: ts-node: command not found
[nodemon] failed to start process, "ts-node ./src/index.ts" exec not found
[nodemon] Error
at Bus. (/usr/local/lib/node_modules/nodemon/lib/nodemon.js:158:25)
at Bus.emit (node:events:526:35)
at ChildProcess. (/usr/local/lib/node_modules/nodemon/lib/monitor/run.js:199:11)
at ChildProcess.emit (node:events:514:28)
at ChildProcess._handle.onexit (node:internal/child_process:294:12)

from export-to-ghostfolio.

Could you perhaps try to run npm install and then run the tool again?

from export-to-ghostfolio.

Thank you! It appears to be processing!

I had to run sudo chown -R 501:20 "/Users/danpennington/.npm" then run npm install again.

Looking great! Now I can play with Ghostfolio a bit. I will report anything that looks unusual.

MANY many thx!

from export-to-ghostfolio.

Question:
I'm not sure where the CUSIP that your converter or Ghostfolio come from, but it doesn't seem to recognize a US T-Bill I have in a new portfolio I was trying to convert.
The "Symbol" Schwab is exporting is: 912797HX8
Which is https://www.treasurydirect.gov/instit/annceresult/press/preanre/2023/A_20230829_2.pdf
No biggie as I don't have many T-Bills.

from export-to-ghostfolio.

What CSV line is that at? 912797HX8 does not return a result when searching on Yahoo Finance.

from export-to-ghostfolio.

What CSV line is that at? 912797HX8 does not return a result when searching on Yahoo Finance.

Thanks for your help Dick!
912797HX8 is in a new CSV I just downloaded. No worries: since I only have a few T-Bills I will remove them from the CSV. Maybe Ghostfolio allows me to add them individually?

from export-to-ghostfolio.

You can add records manually to Ghostfolio, but if Ghostfolio can find it, the tool should as well!

from export-to-ghostfolio.

You can add records manually to Ghostfolio, but if Ghostfolio can find it, the tool should as well!

Thomas suggested something similar:
"One possibility would be to record them manually as a "Valuable", meaning that it does not track the current market value, but the value appears at least in the net worth calculation." so I will try that.

I did have another error - it may have to do with Interest. Please see the attached sample file.

The error it threw was:
throw new ErrorClass(errorObj.description);
^
BadRequestError: Missing required query parameter=q

Any ideas?

Many thx!
Dan
Sample_Transactions.csv

from export-to-ghostfolio.

That one is not processed correctly. I will add an Interest record (I just checked and Ghostfolio does support interest records).

from export-to-ghostfolio.

@DCPDad I just added the interest processing, can you try if it works?

from export-to-ghostfolio.

We are making great headway!

I haven't tried the import process on this new file yet, but I did see these messages after the conversion process:

No result found for dividend action for SNAXX with currency USD! Please add this manuall.
[i] No result found for buy action for QQQ with currency USD! Please add this manually..
[i] No result found for sell action for SNAXX with currency USD! Please add this manually..
[i] No result found for dividend action for SNAXX with currency USD! Please add this manuall.
[i] No result found for sell action for SNAXX with currency USD! Please add this manually..
[i] No result found for dividend action for SNAXX with currency USD! Please add this manuall.

Any thoughts?

Thx-
Dan

from export-to-ghostfolio.

Hello Dick!

It also appears there are several "Invalid date" items in the output (attached).
ghostfolio-schwab.json

from export-to-ghostfolio.

Thanks, will try to look into these issues when I get back from holiday.

from export-to-ghostfolio.

I hope your holidays have been peaceful and calm, Dick.

Have you had a chance to look into this?

Many thx!
Dan

from export-to-ghostfolio.

It was very good, thanks for asking!

I have just pushed some changes which fix the date issues.

As for the other messages you sent, I don't see those when I run the tool with the example export. Can you try to run it again with the latest code, and send me the records that fail if the issue persists? Thanks!

from export-to-ghostfolio.

Glad you had a great holiday.

Thank you for your help.

I am getting this error when I try to run your new update:
Error: No converter provided (i.e. trading212, degiro)

Any thoughts?

from export-to-ghostfolio.

Did you run it with the schwab command, like npm run start schwab?

from export-to-ghostfolio.

Yes. I downloaded your latest, modified the .env, ran "npm install" and then ran "npm run start schwab".

from export-to-ghostfolio.

One more check, you did run the code from here?

from export-to-ghostfolio.

My bad! Sorry about that -- I must have grabbed it from the production code! (It has been a while since I updated).

Progress!

I do now get three errors when trying to import into Ghostfolio:
activities.9.date must be a valid ISO 8601 date string
{
"accountId": "",
"comment": "",
"fee": 0.02,
"quantity": 1,
"type": "INTEREST",
"unitPrice": 0.02,
"currency": "USD",
"dataSource": "MANUAL",
"date": "Invalid Date",
"symbol": "BANK INT 082923-091523 SCHWAB BANK"

from export-to-ghostfolio.

Ah yes, that's my bad! I forgot to put the US date conversion on interest records. Should be fixed now if you update again!

from export-to-ghostfolio.

That's great news! Please let me know how it goes and if you find any inconsistenties that need fixing.

If all goes well, could you provide export instructions like here so I can add those to the manual? Much appreciated!

from export-to-ghostfolio.

So far SO GOOD!

Here are the "instructions" -- I hope this helps!

Schwab
Login to your Schwab account. Go to “Accounts” then “History”. Select the account you want to download details from. Select the “Date Range” and select “Export” (csv). Save the file.

from export-to-ghostfolio.

Great to hear it's working well! And thank you for the instructions, I will add those tomorrow!

from export-to-ghostfolio.

I think I see a small problem (he says hopefully!).

In comparing the CSV import to what Ghostfolio has, maybe a comma got recognized wrong.

The CSV import has this line:
"09/08/2023","Sell","SNAXX","SCHWAB VALUE ADVANTAGE MONEY ULTRA","500,135","$1.00","","$500135.00"

The json export of activities has this for the same transaction:
{
"accountId": "",
"comment": null,
"fee": 0,
"quantity": 500,
"type": "SELL",
"unitPrice": 1,
"currency": "USD",
"dataSource": "YAHOO",
"date": "2023-09-08T07:00:00.000Z",
"symbol": "SNAXX"
},

from export-to-ghostfolio.

That would be an US value parsing issue (since the usage of comma's and dots in numbers is different).

I have changed some code and it now returns the following for me:

{
            "accountId": "aaaaaaaaaaaaaaaaaaaaaaaa",
            "comment": "",
            "fee": 0,
            "quantity": 500.135,
            "type": "SELL",
            "unitPrice": 100,
            "currency": "USD",
            "dataSource": "YAHOO",
            "date": "2023-09-08T00:00:00+02:00",
            "symbol": "SNAXX"
        },

Do these values match what you expect?

PS: I have committed the changes, so when you update the code you can try again yourself as well!

from export-to-ghostfolio.

I really appreciate your help and truly feel we are SO close -- thank you!

I downloaded the latest and get this new error:
TSError: ⨯ Unable to compile TypeScript:
src/converters/schwabConverter.ts:12:14 - error TS18052: Non-abstract class 'SchwabConverter' does not implement all abstract members of 'AbstractConverter'

from export-to-ghostfolio.

Should be fixed now (however I fixed this on my phone so I haven’t tested it 🤞🏼). If it doesn’t work now, I’ll fix it for you tomorrow!

I did some code cleanup this morning but forgot to reflect those changes in the Schwab code.

Edit: found another small typo which I just fixed.

from export-to-ghostfolio.

Hmmm. Looking forward to you taking a look tomorrow. Again, I appreciate your help.

Most recent build got me this error:
npm run start schwab

[email protected] start
nodemon schwab

[nodemon] 3.0.2
[nodemon] to restart at any time, enter rs
[nodemon] watching path(s): src/**/*
[nodemon] watching extensions: ts,js
[nodemon] starting ts-node ./src/index.ts schwab
[i] Processing file using Schwab converter
/Users//Desktop/1-Ghostfolio Test/Export-To-Ghostfolio-feature-Add-Schwab-Converter/src/index.ts:42
throw new Error("No converter provided (i.e. trading212, degiro)");
^
Error: No converter provided (i.e. trading212, degiro)
at Object. (/Users//Desktop/1-Ghostfolio Test/Export-To-Ghostfolio-feature-Add-Schwab-Converter/src/index.ts:42:15)
at Module._compile (node:internal/modules/cjs/loader:1376:14)
at Module.m._compile (/Users//Desktop/1-Ghostfolio Test/Export-To-Ghostfolio-feature-Add-Schwab-Converter/node_modules/ts-node/src/index.ts:1618:23)
at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)
at Object.require.extensions. [as .ts] (/Users//Desktop/1-Ghostfolio Test/Export-To-Ghostfolio-feature-Add-Schwab-Converter/node_modules/ts-node/src/index.ts:1621:12)
at Module.load (node:internal/modules/cjs/loader:1207:32)
at Function.Module._load (node:internal/modules/cjs/loader:1023:12)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:135:12)
at phase4 (/Users//Desktop/1-Ghostfolio Test/Export-To-Ghostfolio-feature-Add-Schwab-Converter/node_modules/ts-node/src/bin.ts:649:14)
at bootstrap (/Users//Desktop/1-Ghostfolio Test/Export-To-Ghostfolio-feature-Add-Schwab-Converter/node_modules/ts-node/src/bin.ts:95:10)
[nodemon] app crashed - waiting for file changes before starting...

from export-to-ghostfolio.

Did you test these changes?

from export-to-ghostfolio.

Ha - no I think I used the one we had been using. Sorry about that - it looks like you have integrated into the main one which is awesome!
I will test and report.
Thank you and Happy New Year!

from export-to-ghostfolio.

Just downloaded it from your primary source (the one you just now sent), and I am getting the same/similar error:

npm run start schwab

[email protected] start
nodemon schwab

[nodemon] 3.0.2
[nodemon] to restart at any time, enter rs
[nodemon] watching path(s): src/**/*
[nodemon] watching extensions: ts,js
[nodemon] starting ts-node ./src/index.ts schwab
[i] Processing file using Schwab converter
/Users/xxx/Desktop/1-Ghostfolio Test/Export-To-Ghostfolio-feature-Add-Schwab-Converter/src/index.ts:42
throw new Error("No converter provided (i.e. trading212, degiro)");

from export-to-ghostfolio.

The one I sent should be the work in progress version of Schwab. I haven't merged it to primary since I want to be sure it works to your satisfaction before I make it available for the public 😄

If that one does not work however, I currently dont know what it could be. I will look into it tomorrow and come back to you.

from export-to-ghostfolio.

@DCPDad I found the issue and have resolved it just now. Whenever you have time, please try again.

from export-to-ghostfolio.

NO errors during run! Yay!

But it looks like something similar is happening to 'fees':
Input file shows:
10/10/2023 Advisor Fee MGMTFEE TO ADVISOR -$2862.87

Output file shows:

from export-to-ghostfolio.

Opps - same with Cash Dividend. It should be $1431.87, but after the conversion, it is 143187.00

from export-to-ghostfolio.

Looks like I accidentally added some code that completely broke the amounts. I removed that and when I run it myself, I get logical values.

from export-to-ghostfolio.

I'm planning to add some automated testing early next year that automatically validates these kind of errors and warns me in advance. This also will help me validate other fixes as I don't have to do it all manually every time 😉

from export-to-ghostfolio.

Ha - thanks and trying some automation is a great idea!!

I honestly do appreciate your back-and-forth with me. I feel we are getting quite close!

Please let me know when you want an amateur to test again :)

from export-to-ghostfolio.

You can run your test again now!

from export-to-ghostfolio.

CLOSER!

Looks like the "Sell" calculation might be the only one still broken:
09/08/2023 Sell SNAXX SCHWAB VALUE ADVANTAGE MONEY ULTRA 500,135 $1.00 $500135.00
shows as:

from export-to-ghostfolio.

That was an issue with numbers > 999, should also be fixed now. That specific record you mentioned now has a quantity of 500135.

from export-to-ghostfolio.

@DCPDad Have you had a chance to look at the results? I would like to promote these changes to the main branch so other users can also use the tool with Schwab 🙂.

from export-to-ghostfolio.

Due to inactivity I have merged the Schwab converter to the main branch. Should there be any issues left, please create an issue so that we can look into resolving it.

from export-to-ghostfolio.