rhysmorgan134 / node-carplay Goto Github PK

#15 needs merging to npm

As discussed in PRs investigation need to determine whether dynamic scaling is possible

Creating a tracking issue, will open a PR for this.

There are quite a few reasons that have led me down this path. For context, I have a car with an AUX port, and a pop-up nav. That's where I am trying to fit my carplay setup.

While there are quite a few upsides to the web version, for example:

• A working audio stack, without popping or crackling
• Low latency video
• No need to install any software besides chrome
• Quite reliable in testing so far

There are also some downsides, which I hit in my usecase

• Slow boot times

The raspberry pi I used for testing (4; model b) was already not super-fast in comparison to regular in-car infotainment systems, but add on to the fact that you have to wait for the desktop environment to get ready (I was using sway) and then for the browser to load (sway config had exec for chrome), and then for the browser to initialize the dongle, the delay adds up.

In contrast, one can have a simple systemd daemon that starts the node.js dongle server as soon as the device boots up, in parallel to initialization of the dongle

• Ability to have "isolated" web apps without losing Carplay state

In my usecase, I have quite a few different paths/pages and it is a lot more convinient to have a built carplay and dump it in a folder and navigate to it when you need to show carplay. It is harder to do it in current config because the dongle gets reset/loses state when you navigate away.

This requires all the other infotainment code to be written in the same repository as the dongle handler, and makes mixing and matching much less practical.

• Audio output/input seems impossible to get "right" if you are using USB capture cards for video

While 3.5mm jack works okay for audio output, audio input on the pi is a bit tricky. I wanted to use a USB audio device to handle input, while having some usb capture cards attached that handle reverse and forward cameras, (and I tested many audio usb dongles from different brands, not sure if with different underlying chipsets) and it was an utterly tiresome, frustrating experience that made me finally understand "At least my macOS has working audio lolx" memes I see all around the internet

I mainly used Raspbian bullseye for my testing, and during testing, Alsa devices would get a random index each time so you couldn't rely on the config to have the same effect each time, disabling/blocklisting some USB devices from audio recording while allowing others was also buggy, even if you got everything right there would just be no sound. So after wasting a few days on it, I just gave up.

My current solution (not yet implemented) is to use a USB Bluetooth to Aux adapter (like this; non-affiliate link). Then I can configure the audio output to box in the carplay dongle config and just offload the entire audio problem to dedicated hardware that works reliably well.

This is not directly related to the web version but just noting a frustration of the platform/environment itself.

With the node version, the idea is to resurrent the PR from old, and yank out the audio (since it'll be going to dongle) and publishing it in case people want to use it.

Is there any interest in this @gozmanyoni @rhysmorgan134?

Edit: Could be interesting to create a bun executable of the node version: https://bun.sh/docs/bundler/executables

Currently there's no package.json#files or .npmignore so the npm package size is >65MB unpacked (in next branch). Let's fix that

...
...
npm notice 219B    examples/carplay-web-app-direct/src/setupProxy.js
npm notice 240B    examples/carplay-web-app-direct/src/setupTests.ts
npm notice 2.7kB   examples/carplay-web-app-direct/src/useCarplayAudio.ts
npm notice 503B    examples/carplay-web-app-direct/tsconfig.json
npm notice 599B    examples/carplay-web-app-worker/config-overrides.js
npm notice 1.4kB   examples/carplay-web-app-worker/package.json
npm notice 3.9kB   examples/carplay-web-app-worker/public/favicon.ico
npm notice 1.7kB   examples/carplay-web-app-worker/public/index.html
npm notice 5.3kB   examples/carplay-web-app-worker/public/logo192.png
npm notice 9.7kB   examples/carplay-web-app-worker/public/logo512.png
npm notice 492B    examples/carplay-web-app-worker/public/manifest.json
npm notice 67B     examples/carplay-web-app-worker/public/robots.txt
npm notice 469B    examples/carplay-web-app-worker/README.md
npm notice 564B    examples/carplay-web-app-worker/src/App.css
npm notice 266B    examples/carplay-web-app-worker/src/App.test.tsx
npm notice 5.3kB   examples/carplay-web-app-worker/src/App.tsx
npm notice 366B    examples/carplay-web-app-worker/src/index.css
npm notice 493B    examples/carplay-web-app-worker/src/index.tsx
npm notice 2.6kB   examples/carplay-web-app-worker/src/logo.svg
npm notice 40B     examples/carplay-web-app-worker/src/react-app-env.d.ts
npm notice 416B    examples/carplay-web-app-worker/src/reportWebVitals.ts
npm notice 219B    examples/carplay-web-app-worker/src/setupProxy.js
npm notice 240B    examples/carplay-web-app-worker/src/setupTests.ts
npm notice 2.7kB   examples/carplay-web-app-worker/src/useCarplayAudio.ts
npm notice 1.4kB   examples/carplay-web-app-worker/src/worker/carplay.ts
npm notice 589B    examples/carplay-web-app-worker/src/worker/types.ts
npm notice 503B    examples/carplay-web-app-worker/tsconfig.json
npm notice 230B    jest.config.js
npm notice 1.7kB   package.json
npm notice 1.1kB   scripts/configIcon.ts
npm notice 352B    scripts/startnode.ts
npm notice 8.4kB   src/modules/__tests__/DongleDriver.test.ts
npm notice 12.2kB  src/modules/__tests__/messages.test.ts
npm notice 1.6kB   src/modules/__tests__/mocks/usbMocks.ts
npm notice 6.2kB   src/modules/DongleDriver.ts
npm notice 103B    src/modules/index.ts
npm notice 16.0kB  src/modules/messages.ts
npm notice 117B    src/modules/utils.ts
npm notice 4.2kB   src/node/CarplayNode.ts
npm notice 156B    src/node/index.ts
npm notice 940B    src/node/NodeMicrophone.ts
npm notice 3.1kB   src/web/CarplayWeb.ts
npm notice 180B    src/web/index.ts
npm notice 1.4kB   src/web/recorder.worklet.ts
npm notice 207B    src/web/tsconfig.json
npm notice 1.5kB   src/web/WebMicrophone.ts
npm notice 115B    tsconfig.build.json
npm notice 397B    tsconfig.json
npm notice === Tarball Details ===
npm notice name:          node-carplay
npm notice version:       4.0.0
npm notice filename:      node-carplay-4.0.0.tgz
npm notice package size:  14.7 MB
npm notice unpacked size: 65.5 MB
npm notice shasum:        45b5787692dd0c5600d5c9a89b5f72bc304ed96b
npm notice integrity:     sha512-/oyscJBGG9sy7[...]BGpkf1ZbEPdIA==
npm notice total files:   1216
npm notice
node-carplay-4.0.0.tgz

Modern NPM runs prepare script automatically after install and it's recommended to put your build logic in there

It also has the added benefit of being triggered before publish, making sure we don't ever publish stale builds.

See https://docs.npmjs.com/cli/v10/using-npm/scripts#life-cycle-scripts

Hi, I am working on implementing marcdubois71450/tesla-carplay that is forked from your project if I recall. I know that is not your project, but I was wondering if you might have some answers that are related to node-CarPlay.

I have set the canvas size to 1180x960 and now I am seeing the album and track name fonts being scaled too big. Do you have any idea what values set these items? I believe these values are set in jsmpeg.min.js. See my attached screenshot showing the oversized fonts.

On a separate issue, I am also trying to determine where the touch overlay map is that sets where the touch buttons are on the screen. Not all my touch locations line up and it seems to also be resolution related.

Not sure if you would know either of my questions, but thank you in advance.

Hi,
is multitouch actually works?

I tried in Maps app, but it works like single touch.

We should document known issues somewhere, not sure what's the best place to do so

I'd start with this one:

• Chrome has a USBDevice.reset bug on macOS that it doesn't actually work (https://bugs.chromium.org/p/chromium/issues/detail?id=1189418)

Hence I was getting inconsistent reset experience in node vs browser

I used the integrated configIcon.ts script and all I'm getting is what's in the screenshot below.

Is there an example icon file that works fine?

I have a react web app that uses node-carplay for Apple Carplay.

Before multitouch was released everyhing worked fine. I wanted to upgrade to the latest version now but I can't interact with Carplay anymore. I'm no web-developer so it's a little bit harder for me to trace down the issue, also since everything is packed up now a bit differently.

When running the web-app example it works, not in my own app though. I changed the location of some files a bit, updated the imports but besides that I'm using 1:1 the same codebase. I loose the trace after the useCarplayTouch.ts file. In there it's calling sendTouchEvent which should send the touchEvent to the node-carplay module which then sends it to the dongle.

I can see that sendTouchEvent is being called. When I console.log in the sendable.js from the node_modules folder, I can see a log in the web-app example. In my app I don't see the log at the same spot. I'm not sure what could cause this. A wrong tsconfig or vite.config file?

I made sure to meticulously copy over every change from the latest release. Stream and everything works fine, I just can't click or "touch" anything when carplay has launched.

If I can add filecontents or any other information please let me know.

CarPlay supports messages to be sent from the car for items such as fuel level, battery percent, speed etc. I am hoping these are almost a "pass through" interface on the dongle, so whilst maybe not implemented in a verbose way in the APK, maybe when you send a message with the right format this gets relayed onto CarPlay direct. Just a suspicion at this stage, but perhaps some Apple documentation along with the APK may be the key to this one.

Looks like it's possible to port whole project to browser (chrome) with WebUSB to communicate with usb dongle and decode video with WEB Codecs or any other js demuxer.
I have two CarPlay dongles, not Android ones, will they work?

Hi Rhys,

This is an excellent project and just what I have been looking for. However, am I correct in thinking that microphone input for calls is currently not working? I experimented with changing some of the code in AudioParse.js (based on the protocol description here https://github.com/electric-monk/pycarplay/blob/8d9a31370546515991260b4056c1650e9e097755/protocol.py#L262) to see if I could get it to work (handle type 4 and 5), but so far I haven't had any luck.

Siri is working perfectly, however.

In my application which uses the code of the CarPlay web-app example, I'm getting the above error in the Render.worker when using the latest chromium release (124).

I downgraded back to 116 which was the version I used prior to the update and it worked again.

Just sticking to 116 for know but is that a known issue?

Best regards

I've broken jest at least a couple of times now, I am thinking it's time to setup CI. and while we're at it, we could also store the generated example's output, so people can just download that zip and run it directly, instead of having to know download/install Node.js etc.

Hello and thank you so much for your project; it has been very helpful to me. I've encountered an issue with the default autokit app on the main unit in my car - it consistently loses connection. Using additional applications, I noticed that the device disconnects and reconnects to USB, leading me to initially think the problem was with the hardware of my head unit.

Then I came across your project, ran it on my Mac and Raspberry Pi, and conducted numerous experiments. I appreciate your algorithm, the implementation of the protocol, and the comments, which have greatly helped me understand how the adapter works.

I noticed that the behavior of the adapter when working with your code matches its behavior when using the original autokit app. It sometimes disconnects and may not connect on the first attempt. This indicates that a more suitable approach for me would be to write my own Android app and use it instead of the standard autokit. I have done this, even implemented it in Flutter, and it can be tested on both Mac and Android (I haven't tested it on other platforms, but I don't anticipate any issues).

At the moment, it only implements the protocol (I have ported your classes for working with the adapter, but haven't implemented video output yet; that will come a bit later). However, I've encountered some issues. Your code works perfectly and connects to the device every time, but my code (which I'm testing on a Mac) works up to a certain point and then loses connection. I can't figure out the reason. I've provided logs with binary data and message names; perhaps you can help me understand what's going wrong?

Repo available here ->
abuharsky/carplay#1

Currently we use console.debug and such in places around the codebase. It'll be better to use debug to standardize the logging, and allow people to selectively enable logging for different components of the codebase.

Originating discussion happened here #20 (comment)

I need to use it in headless mode, I want to connect to server over wifi, so, can you add ability to change webusb to local usb?

Hey Rhys,

Firstly i would like to thank for your great work, i'm going to use your apps in my project in a car. I have a joystick there, and i would like to bind some new keys - like shortcut to messages or shortcut to google maps
I saw there are bindings in react app but it leads to key bindings in node-carplay -> DongleHandler.
There is a list of values that equals siri, left, right etc.
Where did you found those values, are they from Autokit.apk ? Or you just wrote something else, that i couldn't ffind ?

I will be grateful for any answer.

Hi,

is it possible to access any navigator data?
"next turn" or something?

Logging detects unknown messages with type 35, 36, etc.
Maybe it is possible to send "Enable navigator events" message?

So I really need it to re-send navigator events to the car instrument cluster.

The dongles have this capability, and I think it's just a case of a different set up config.

I've only just picked up on this as I tend to use the navigation without audio directions. However it seems when navigation audio plays it is feeding into the same audio player, but with a different initial frequency. I have added a prompt whenever a new audio format is found, and music is playing at 48khz 2 channel. When a navigation direction begins, I get a new audio player, at 44.1khz 2 channels.

All seems fine.

However when the audio packets are received for nav, they have the decode type set to 4. As audio is already playing at 48khz it creates mass distortion.

I am wondering whether my dongle version contains a bug or does anyone else see this?

Secondly, I wonder whether we should create a second set of audio players to cover the case of when nav sample rate is the same as music? I am happy to take this on, but just want to ensure it's not a bug on my firmware, as regardless the decodeType received on the first packet does not match the following packets. (I have verified and audio is slowed down when the nav is forced to play at 44.1khz, its not a huge amount, so not easily discernable)

@gozmanyoni @steelbrain

When plugging in an Android phone into the carlinkit adapter with this image (which uses node-carplay), the android auto projection starts up but the screen flickers, this seems to be on the same cycle as the "requesting key frame" messages. The CarPlay projection does not seem to be flickering.

Here is a video where you can see it happening. https://youtu.be/hBsnDfqHGdA (Flickering starts at 0:10, you can see the logs at 0:40)

Do you happen to know what causes this? I have also tested this on version 1.0.5 of react-carplay and the screen does not seem to flicker on that release.

I've added a console.log statement in the componentDidMount() function to test whether the component but this does not seem to be the case. This function is only fired on opening the carplay page, not when plugging in a phone or while the screen is flickering.

If you need any more info please let me know.

Running this on a Raspberry Pi 4B.

Opening a tracking issue for releasing the current changeset in next as a semver-major release.

I've been daily driving it (the web version), and it's working pretty good for me.

I think it's good enough to be released to general public. Any shortcomings can be addressed in follow-up releases.

What do you think? @gozmanyoni @rhysmorgan134

We should have an API so consumers can subscribe to such errors (and attempt to recover by reloading the page, resetting the usb device etc)

web outputs files with ESM exports, while node outputs files with CommonJS exports. Both do not mix when node imports a modules file outside of a bundler/transpiler

This is in the next branch

Tracking issue for the discussion that originated in #40 -- Let's enable wiki in the repository so we can start documenting relevant stuff, and even link to it from the README

This PR is about the new version of node-carplay in next branch for web version. Currently, even tho .reset() is called on the USB interface, it doesn't fully reset and keeps playing music if you reload the page with it.

What that means is that the video comes in, the audio comes in, but Plugged message does not. So there's two ways to handle this:

• Accept the dongle in its current state, and just mark the device as plugged if we're already getting audio and video
• Reset the dongle properly so we go through the expected sequence of events

cc @gozmanyoni

Edit: The implications here would be that we don't send the pair message after X seconds of boot

Hi,

Autokit app has "Software and Hardware decoding method" in settins.
By default it sewts 25 fps for software method, and up to 60 fps for hardware, what does it mean? How to enable hardware?

Is it relative to dongle or to client app that decodes raw h264 frames?

How do you feel about adding code style and linting? More specifically, setting up Prettier and ESLint and doing file cleanups based on those recommendations.

Hey!

I am wondering if you know how to configure the audio source in the config. The AutoKit app gives an option to switch from Box to Bluetooth (in this option the phone sends the bt audio straight to the car/headunit via bt and only video is used in CarPlay).

Best regards

Hi, for some reason video not playing from node app on my car Head Unit browser.

Is it possible to stream it as sequense of images, that can be rendered inside <image src...?

what type of data inside message with type "Video"?

Just opening an issue to gauge thoughts/interests in this. I think pnpm is nice, but for such few dependencies, and for an open source package, it's one extra thing contributors have to install.

What do you think about replacing it with plain npm?

To reproduce: Play some music, look at the timestamp, minimize the tab, open again after 10-15 seconds and the video will be playing catch up instead of showing real time data