Building gasket-dkms broken on linux kernel 6.5.0 (Ubuntu 23.10) about edgetpu HOT 37 CLOSED

@themana

sudo apt update
sudo apt upgrade
sudo apt install devscripts debhelper -y

Then clone and the rest

❯ git clone https://github.com/google/gasket-driver.git
Cloning into 'gasket-driver'...
...
❯ cd gasket-driver; debuild -us -uc -tc -b; cd ..
...
dpkg-deb: building package 'gasket-dkms' in '../gasket-dkms_1.0-18_all.deb'.
...
❯ ls -l gasket-dkms*
-rw-r--r-- 1 dewet dewet 49000 Jan 17 13:17 gasket-dkms_1.0-18_all.deb
-rw-r--r-- 1 dewet dewet  1788 Jan 17 13:18 gasket-dkms_1.0-18_amd64.build
-rw-r--r-- 1 dewet dewet  5642 Jan 17 13:17 gasket-dkms_1.0-18_amd64.buildinfo
-rw-r--r-- 1 dewet dewet  1017 Jan 17 13:17 gasket-dkms_1.0-18_amd64.changes

You can install that .deb on any systems that need to build the kernel module, and it will be rebuilt automatically with newer kernel packages being installed:

❯ sudo dpkg -i gasket-dkms_1.0-18_all.deb
...
Setting up gasket-dkms (1.0-18) ...
...
Building for 6.5.0-14-generic
Building initial module for 6.5.0-14-generic
...
depmod...
Time: 0h:00m:10s
❯ sudo modprobe apex
❯ lsmod | grep apex
apex                   28672  0
gasket                135168  1 apex

from edgetpu.

Stopped working here too. How to build the correct package for kernel 6.5?

@Mikescotland As documented in the earlier notes:

❯ sudo apt install devscripts debhelper dh-dkms -y
...
❯ git clone https://github.com/google/gasket-driver.git
Cloning into 'gasket-driver'...
...
❯ cd gasket-driver; debuild -us -uc -tc -b; cd ..
...
dpkg-deb: building package 'gasket-dkms' in '../gasket-dkms_1.0-18_all.deb'.
...
❯ ls -l gasket-dkms*
-rw-r--r-- 1 dewet dewet 49000 Jan 17 13:17 gasket-dkms_1.0-18_all.deb
-rw-r--r-- 1 dewet dewet  1788 Jan 17 13:18 gasket-dkms_1.0-18_amd64.build
-rw-r--r-- 1 dewet dewet  5642 Jan 17 13:17 gasket-dkms_1.0-18_amd64.buildinfo
-rw-r--r-- 1 dewet dewet  1017 Jan 17 13:17 gasket-dkms_1.0-18_amd64.changes

You can install that .deb on any systems that need to build the kernel module, and it will be rebuilt automatically with newer kernel packages being installed:

❯ sudo dpkg -i gasket-dkms_1.0-18_all.deb
...
Setting up gasket-dkms (1.0-18) ...
...
Building for 6.5.0-14-generic
Building initial module for 6.5.0-14-generic
...
depmod...
Time: 0h:00m:10s
❯ sudo modprobe apex
❯ lsmod | grep apex
apex                   28672  0
gasket                135168  1 apex

from edgetpu.

When will this be available via https://packages.cloud.google.com/apt ?

It's been fixed but still not available in stable for over 3 months.

I really not want to install all the dev dependencies on a production node.

from edgetpu.

FYI Workaround for Proxmox (debian based) systems here: https://forum.proxmox.com/threads/update-error-with-coral-tpu-drivers.136888/#post-608975

from edgetpu.

Make sure you have the dkms debhelper package installed: sudo apt install dh-dkms

Some Proxmox instances require more effort ;)

sudo apt install dh-dkms devscripts git

from edgetpu.

This was fixed a few months ago in https://github.com/google/gasket-driver/ already; for my use case I just built the .deb from that repo directly and deployed across my Coral-bearing machines.

from edgetpu.

This was fixed a few months ago in https://github.com/google/gasket-driver/ already; for my use case I just built the .deb from that repo directly and deployed across my Coral-bearing machines.

I am trying to build it for Promox, can you guide me on how to build it ?

from edgetpu.

This bug is now appearing in the HWE kernel in Ubuntu 22.04 as well. The "stable" repo isn't seeming very table anymore.

What do we need to do to get it packaged and updated in the Debian stable repo?

If you give me write access to the Google Cloud repos, I'll gladly update it myself.

from edgetpu.

Thanks @alienatedsec and @dewet22 this worked great.
Not sure why they havent updated to stable though. Sure would make things much easier

from edgetpu.

Make sure you have the dkms debhelper package installed: sudo apt install dh-dkms

Some Proxmox instances require more effort ;)

sudo apt install dh-dkms devscripts git

Yep , I did post in Double take discord server

from edgetpu.

@alienatedsec That worked perfectly, thank you so much

from edgetpu.

Thank you a ton @dewet22 and @alienatedsec, that did the trick! Very lucky I found this thread after a good deal of hunting. Cheers!

from edgetpu.

Can I just ask @alienatedsec @bbccdd @arevindh if your Proxmox installs are still working with your Coral?

I have:

1. Built gasket-dkms from source as per above
2. Tried reverting to a previous kernel (eg. 6.2.16-20-pve)

Modules seem loaded:

apex                   28672  0
gasket                135168  1 apex

No dice...

~ # ls /dev/apex*
zsh: no matches found: /dev/apex*

I've gone so far as to order another Coral today as I cannot even get this thing recognised and flashed by webcoral now. Starting to suspect a faulty unit.

from edgetpu.

Can I just ask @alienatedsec @bbccdd @arevindh if your Proxmox installs are still working with your Coral?

Sorry for the late response @bobmarley2021 - I don't know why I missed this message - regardless, my proxmox cluster and all three nodes have the latest versions and work just fine. I documented my effort in this comment for frigate installation on proxmox with Coral support

Installing Coral drivers - go to the shell of the host and type the following commands

apt update
apt upgrade
apt install pve-headers-$(uname -r)
apt install proxmox-default-headers
apt install dh-dkms devscripts git
git clone  https://github.com/google/gasket-driver
cd gasket-driver
debuild -us -uc -tc -b -d
cd ..
dpkg -i gasket-dkms_1.0-18_all.deb
reboot

Once rebooted, ensure the apex_0 device is in /dev/.

ls -alh /dev/apex_0 
crw-rw---- 1 root root 120, 0 Mar 20 14:51 /dev/apex_0

Any future kernel updates should automatically include gasket-dkms after above steps.

Follow the rest here blakeblackshear/frigate#5448 (comment)

from edgetpu.

Is there any apex? It doesn't have to be apex_0 for dual edge and it's likely you will see only one. @nglessner

from edgetpu.

Thank you @alienatedsec ! My issue was that I had added it to the blocklist. I eventually got it working!

from edgetpu.

Previously reported as #695 and #785.

from edgetpu.

Are you satisfied with the resolution of your issue?
Yes
No

from edgetpu.

I am trying to build it for Promox, can you guide me on how to build it ?

I don't use or know Proxmox so I can't help directly. If you need help on building a deb from sources, there are plenty of guides on the internet for that and the starting clue is in the linked repo already: debuild is the tool you'll need to use.

from edgetpu.

I am trying to build it for Promox, can you guide me on how to build it ?

I don't use or know Proxmox so I can't help directly. If you need help on building a deb from sources, there are plenty of guides on the internet for that and the starting clue is in the linked repo already: debuild is the tool you'll need to use.

Oh ok , when i try debuild only thing i get is the below

dh: error: unable to load addon dkms: Can't locate Debian/Debhelper/Sequence/dkms.pm in @INC (you may need to install the Debian::Debhelper::Sequence::dkms module)

Debian::Debhelper::Sequence::dkms module) also impossible to install

Did you get any similar errors?

from edgetpu.

Make sure you have the dkms debhelper package installed: sudo apt install dh-dkms

from edgetpu.

dh-dkms

Oops I didnt install dh-dkms , Thank you, let me try building

from edgetpu.

@dewet22 frigate is working now 👍.

from edgetpu.

I have similar issue and the steps above dont work.
If I try the debuild command, it returns
debuild: fatal error at line 679: cannot find readable debian/changlog anywhere!

dh-dkms is installed.

from edgetpu.

I have similar issue and the steps above dont work. If I try the debuild command, it returns debuild: fatal error at line 679: cannot find readable debian/changlog anywhere!

dh-dkms is installed.

Did you make sure you were in the respective directory? Building the gasket-driver should not fail if the necessary prereqs are installed, and you are in the right directory. No changelog error indicates you are not in the right directory.

from edgetpu.

When will this be available via https://packages.cloud.google.com/apt ?

It's been fixed but still not available in stable for over 3 months.

I really not want to install all the dev dependencies on a production node.

Perhaps you could build the package in a Debian LXC or VM, transfer the compiled Debian package to the host Proxmox machine, and then install to avoid installing dev dependencies on a production node? Hope that helps!

from edgetpu.

Stopped working here too. How to build the correct package for kernel 6.5?

from edgetpu.

Got it working
frigate.detectors.plugins.edgetpu_tfl INFO : TPU found

Thanks for your help!

from edgetpu.

@dewet22 I'm trying to follow your directions but I get an error when trying to install dh-dkms, any help would be appreciated.

E: Unable to locate package dh-dkms

This is on the Ubuntu VM with the version info below
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy

from edgetpu.

Just commenting here to say thanks team. I was tearing my hair out with my Coral failing until I found this thread.

from edgetpu.

maybe you should check aspm feature in bios. this thing controls power management of pcie. it helped for me

from edgetpu.

Can I just ask @alienatedsec @bbccdd @arevindh if your Proxmox installs are still working with your Coral?

Sorry for the late response @bobmarley2021 - I don't know why I missed this message - regardless, my proxmox cluster and all three nodes have the latest versions and work just fine. I documented my effort in this comment for frigate installation on proxmox with Coral support

Installing Coral drivers - go to the shell of the host and type the following commands

apt update
apt upgrade
apt install pve-headers-$(uname -r)
apt install proxmox-default-headers
apt install dh-dkms devscripts git
git clone  https://github.com/google/gasket-driver
cd gasket-driver
debuild -us -uc -tc -b -d
cd ..
dpkg -i gasket-dkms_1.0-18_all.deb
reboot

Once rebooted, ensure the apex_0 device is in /dev/.

ls -alh /dev/apex_0 
crw-rw---- 1 root root 120, 0 Mar 20 14:51 /dev/apex_0

Any future kernel updates should automatically include gasket-dkms after above steps.

Follow the rest here blakeblackshear/frigate#5448 (comment)

I wish I could say following your directions worked.

I'm on proxmox 8.1.10 kernel 6.5.13-5 using the dual edge PCIe E-key. As far as I can tell building the gasket driver was successful, however in the end ls -alh /dev/apex_0 results in ls: cannot access '/dev/apex_0': No such file or directory.

running lspci -nn | grep 089a does result in
03:00.0 System peripheral [0880]: Global Unichip Corp. Coral Edge TPU [1ac1:089a] so I believe the system can see the TPU, but for whatever reason isn't able to run the driver?

Anyway, I appreciate the help you've given for this setup.

from edgetpu.

@alienatedsec No. Sorry, I should have been more specific. ls -alh /dev/apex* also returns dev/apex_0': No such file or directory

I wouldn't think this would be the limiting factor, but I am attempting this on a Beelink EQ12 Pro (N305). The only available M.2 slot is the "WiFi" slot that is also CNVi. I had high hopes when I could see the device in lspci, but now I'm not so sure.

interestingly lspci -k shows the device as:
03:00.0 System peripheral: Global Unichip Corp. Coral Edge TPU
Subsystem: Global Unichip Corp. Coral Edge TPU
Kernel driver in use: vfio-pci
Kernel modules: apex

from edgetpu.

@nglessner Did you already set the PCI pass through and added it to the blocklist?

Asking because if you added an apex device to a blocklist, it will not show on the host.

The only available M.2 slot is the "WiFi" slot that is also CNVi.

Most of those slots are CNVi, but have only one PCIe lane, which is the limiting factor to recognise only one Coral.

from edgetpu.

@alienatedsec not a problem - thanks for replying. I have since solved the issue on my end. In my case there was part faulty product and part user error (although the user error wouldn’t have mattered at that point due to the former).

I am using a USB coral and didn’t realise that apex is only for PCIe coral. The fault I had with my USB device was causing it not to flash firmware on connection. When I first encountered this problem I googled and was barking up the wrong tree following gasket driver instructions. Then I read on some reviews that certain batches of the USB devices go faulty and appear to exhibit this behaviour - the only solution being to RMA. I replaced the unit and now all is well. I pinned an earlier kernel to be safe, reinstalled the USB driver from the official google repo and I have been up and running ever since :-) 🎉

from edgetpu.

Most of those slots are CNVi, but have only one PCIe lane, which is the limiting factor to recognise only one Coral.

Is it normal for the CNVi slot to not even recognize the Coral?

I've got an HP Elite Mini, and I can't even see the card in lspci in Proxmox.
I can't decide if it's broken, or if something is actually misconfigured in the BIOS.

More likely, it's just not supported in the BIOS.

from edgetpu.

ok with : Linux proxmox 6.8.4-2-pve

after recomplication of Coral driver // gasket-dkms_1.0-18_all.deb

from edgetpu.