softroce / rxe-dev Goto Github PK

Development Repository for RXE

License: Other

Makefile 0.28% C 96.65% Assembly 2.01% C++ 0.56% Objective-C 0.28% Shell 0.03% Perl 0.11% Clojure 0.01% Awk 0.01% Python 0.03% UnrealScript 0.01% Scilab 0.01% Yacc 0.02% Lex 0.01% SourcePawn 0.01%

rxe-dev's Introduction

        Linux kernel release 3.x <http://kernel.org/>

These are the release notes for Linux version 3.  Read them carefully,
as they tell you what this is all about, explain how to install the
kernel, and what to do if something goes wrong. 

WHAT IS LINUX?

  Linux is a clone of the operating system Unix, written from scratch by
  Linus Torvalds with assistance from a loosely-knit team of hackers across
  the Net. It aims towards POSIX and Single UNIX Specification compliance.

  It has all the features you would expect in a modern fully-fledged Unix,
  including true multitasking, virtual memory, shared libraries, demand
  loading, shared copy-on-write executables, proper memory management,
  and multistack networking including IPv4 and IPv6.

  It is distributed under the GNU General Public License - see the
  accompanying COPYING file for more details. 

ON WHAT HARDWARE DOES IT RUN?

  Although originally developed first for 32-bit x86-based PCs (386 or higher),
  today Linux also runs on (at least) the Compaq Alpha AXP, Sun SPARC and
  UltraSPARC, Motorola 68000, PowerPC, PowerPC64, ARM, Hitachi SuperH, Cell,
  IBM S/390, MIPS, HP PA-RISC, Intel IA-64, DEC VAX, AMD x86-64, AXIS CRIS,
  Xtensa, Tilera TILE, AVR32 and Renesas M32R architectures.

  Linux is easily portable to most general-purpose 32- or 64-bit architectures
  as long as they have a paged memory management unit (PMMU) and a port of the
  GNU C compiler (gcc) (part of The GNU Compiler Collection, GCC). Linux has
  also been ported to a number of architectures without a PMMU, although
  functionality is then obviously somewhat limited.
  Linux has also been ported to itself. You can now run the kernel as a
  userspace application - this is called UserMode Linux (UML).

DOCUMENTATION:

 - There is a lot of documentation available both in electronic form on
   the Internet and in books, both Linux-specific and pertaining to
   general UNIX questions.  I'd recommend looking into the documentation
   subdirectories on any Linux FTP site for the LDP (Linux Documentation
   Project) books.  This README is not meant to be documentation on the
   system: there are much better sources available.

 - There are various README files in the Documentation/ subdirectory:
   these typically contain kernel-specific installation notes for some 
   drivers for example. See Documentation/00-INDEX for a list of what
   is contained in each file.  Please read the Changes file, as it
   contains information about the problems, which may result by upgrading
   your kernel.

 - The Documentation/DocBook/ subdirectory contains several guides for
   kernel developers and users.  These guides can be rendered in a
   number of formats:  PostScript (.ps), PDF, HTML, & man-pages, among others.
   After installation, "make psdocs", "make pdfdocs", "make htmldocs",
   or "make mandocs" will render the documentation in the requested format.

INSTALLING the kernel source:

 - If you install the full sources, put the kernel tarball in a
   directory where you have permissions (eg. your home directory) and
   unpack it:

     gzip -cd linux-3.X.tar.gz | tar xvf -

   or

     bzip2 -dc linux-3.X.tar.bz2 | tar xvf -

   Replace "X" with the version number of the latest kernel.

   Do NOT use the /usr/src/linux area! This area has a (usually
   incomplete) set of kernel headers that are used by the library header
   files.  They should match the library, and not get messed up by
   whatever the kernel-du-jour happens to be.

 - You can also upgrade between 3.x releases by patching.  Patches are
   distributed in the traditional gzip and the newer bzip2 format.  To
   install by patching, get all the newer patch files, enter the
   top level directory of the kernel source (linux-3.X) and execute:

     gzip -cd ../patch-3.x.gz | patch -p1

   or

     bzip2 -dc ../patch-3.x.bz2 | patch -p1

   Replace "x" for all versions bigger than the version "X" of your current
   source tree, _in_order_, and you should be ok.  You may want to remove
   the backup files (some-file-name~ or some-file-name.orig), and make sure
   that there are no failed patches (some-file-name# or some-file-name.rej).
   If there are, either you or I have made a mistake.

   Unlike patches for the 3.x kernels, patches for the 3.x.y kernels
   (also known as the -stable kernels) are not incremental but instead apply
   directly to the base 3.x kernel.  For example, if your base kernel is 3.0
   and you want to apply the 3.0.3 patch, you must not first apply the 3.0.1
   and 3.0.2 patches. Similarly, if you are running kernel version 3.0.2 and
   want to jump to 3.0.3, you must first reverse the 3.0.2 patch (that is,
   patch -R) _before_ applying the 3.0.3 patch. You can read more on this in
   Documentation/applying-patches.txt

   Alternatively, the script patch-kernel can be used to automate this
   process.  It determines the current kernel version and applies any
   patches found.

     linux/scripts/patch-kernel linux

   The first argument in the command above is the location of the
   kernel source.  Patches are applied from the current directory, but
   an alternative directory can be specified as the second argument.

 - Make sure you have no stale .o files and dependencies lying around:

     cd linux
     make mrproper

   You should now have the sources correctly installed.

SOFTWARE REQUIREMENTS

   Compiling and running the 3.x kernels requires up-to-date
   versions of various software packages.  Consult
   Documentation/Changes for the minimum version numbers required
   and how to get updates for these packages.  Beware that using
   excessively old versions of these packages can cause indirect
   errors that are very difficult to track down, so don't assume that
   you can just update packages when obvious problems arise during
   build or operation.

BUILD directory for the kernel:

   When compiling the kernel, all output files will per default be
   stored together with the kernel source code.
   Using the option "make O=output/dir" allow you to specify an alternate
   place for the output files (including .config).
   Example:

     kernel source code: /usr/src/linux-3.X
     build directory:    /home/name/build/kernel

   To configure and build the kernel, use:

     cd /usr/src/linux-3.X
     make O=/home/name/build/kernel menuconfig
     make O=/home/name/build/kernel
     sudo make O=/home/name/build/kernel modules_install install

   Please note: If the 'O=output/dir' option is used, then it must be
   used for all invocations of make.

CONFIGURING the kernel:

   Do not skip this step even if you are only upgrading one minor
   version.  New configuration options are added in each release, and
   odd problems will turn up if the configuration files are not set up
   as expected.  If you want to carry your existing configuration to a
   new version with minimal work, use "make oldconfig", which will
   only ask you for the answers to new questions.

 - Alternative configuration commands are:

     "make config"      Plain text interface.

     "make menuconfig"  Text based color menus, radiolists & dialogs.

     "make nconfig"     Enhanced text based color menus.

     "make xconfig"     X windows (Qt) based configuration tool.

     "make gconfig"     X windows (Gtk) based configuration tool.

     "make oldconfig"   Default all questions based on the contents of
                        your existing ./.config file and asking about
                        new config symbols.

     "make silentoldconfig"
                        Like above, but avoids cluttering the screen
                        with questions already answered.
                        Additionally updates the dependencies.

     "make olddefconfig"
                        Like above, but sets new symbols to their default
                        values without prompting.

     "make defconfig"   Create a ./.config file by using the default
                        symbol values from either arch/$ARCH/defconfig
                        or arch/$ARCH/configs/${PLATFORM}_defconfig,
                        depending on the architecture.

     "make ${PLATFORM}_defconfig"
                        Create a ./.config file by using the default
                        symbol values from
                        arch/$ARCH/configs/${PLATFORM}_defconfig.
                        Use "make help" to get a list of all available
                        platforms of your architecture.

     "make allyesconfig"
                        Create a ./.config file by setting symbol
                        values to 'y' as much as possible.

     "make allmodconfig"
                        Create a ./.config file by setting symbol
                        values to 'm' as much as possible.

     "make allnoconfig" Create a ./.config file by setting symbol
                        values to 'n' as much as possible.

     "make randconfig"  Create a ./.config file by setting symbol
                        values to random values.

     "make localmodconfig" Create a config based on current config and
                           loaded modules (lsmod). Disables any module
                           option that is not needed for the loaded modules.

                           To create a localmodconfig for another machine,
                           store the lsmod of that machine into a file
                           and pass it in as a LSMOD parameter.

                   target$ lsmod > /tmp/mylsmod
                   target$ scp /tmp/mylsmod host:/tmp

                   host$ make LSMOD=/tmp/mylsmod localmodconfig

                           The above also works when cross compiling.

     "make localyesconfig" Similar to localmodconfig, except it will convert
                           all module options to built in (=y) options.

   You can find more information on using the Linux kernel config tools
   in Documentation/kbuild/kconfig.txt.

 - NOTES on "make config":

    - Having unnecessary drivers will make the kernel bigger, and can
      under some circumstances lead to problems: probing for a
      nonexistent controller card may confuse your other controllers

    - Compiling the kernel with "Processor type" set higher than 386
      will result in a kernel that does NOT work on a 386.  The
      kernel will detect this on bootup, and give up.

    - A kernel with math-emulation compiled in will still use the
      coprocessor if one is present: the math emulation will just
      never get used in that case.  The kernel will be slightly larger,
      but will work on different machines regardless of whether they
      have a math coprocessor or not.

    - The "kernel hacking" configuration details usually result in a
      bigger or slower kernel (or both), and can even make the kernel
      less stable by configuring some routines to actively try to
      break bad code to find kernel problems (kmalloc()).  Thus you
      should probably answer 'n' to the questions for "development",
      "experimental", or "debugging" features.

COMPILING the kernel:

 - Make sure you have at least gcc 3.2 available.
   For more information, refer to Documentation/Changes.

   Please note that you can still run a.out user programs with this kernel.

 - Do a "make" to create a compressed kernel image. It is also
   possible to do "make install" if you have lilo installed to suit the
   kernel makefiles, but you may want to check your particular lilo setup first.

   To do the actual install, you have to be root, but none of the normal
   build should require that. Don't take the name of root in vain.

 - If you configured any of the parts of the kernel as `modules', you
   will also have to do "make modules_install".

 - Verbose kernel compile/build output:

   Normally, the kernel build system runs in a fairly quiet mode (but not
   totally silent).  However, sometimes you or other kernel developers need
   to see compile, link, or other commands exactly as they are executed.
   For this, use "verbose" build mode.  This is done by inserting
   "V=1" in the "make" command.  E.g.:

     make V=1 all

   To have the build system also tell the reason for the rebuild of each
   target, use "V=2".  The default is "V=0".

 - Keep a backup kernel handy in case something goes wrong.  This is 
   especially true for the development releases, since each new release
   contains new code which has not been debugged.  Make sure you keep a
   backup of the modules corresponding to that kernel, as well.  If you
   are installing a new kernel with the same version number as your
   working kernel, make a backup of your modules directory before you
   do a "make modules_install".

   Alternatively, before compiling, use the kernel config option
   "LOCALVERSION" to append a unique suffix to the regular kernel version.
   LOCALVERSION can be set in the "General Setup" menu.

 - In order to boot your new kernel, you'll need to copy the kernel
   image (e.g. .../linux/arch/i386/boot/bzImage after compilation)
   to the place where your regular bootable kernel is found. 

 - Booting a kernel directly from a floppy without the assistance of a
   bootloader such as LILO, is no longer supported.

   If you boot Linux from the hard drive, chances are you use LILO, which
   uses the kernel image as specified in the file /etc/lilo.conf.  The
   kernel image file is usually /vmlinuz, /boot/vmlinuz, /bzImage or
   /boot/bzImage.  To use the new kernel, save a copy of the old image
   and copy the new image over the old one.  Then, you MUST RERUN LILO
   to update the loading map!! If you don't, you won't be able to boot
   the new kernel image.

   Reinstalling LILO is usually a matter of running /sbin/lilo. 
   You may wish to edit /etc/lilo.conf to specify an entry for your
   old kernel image (say, /vmlinux.old) in case the new one does not
   work.  See the LILO docs for more information. 

   After reinstalling LILO, you should be all set.  Shutdown the system,
   reboot, and enjoy!

   If you ever need to change the default root device, video mode,
   ramdisk size, etc.  in the kernel image, use the 'rdev' program (or
   alternatively the LILO boot options when appropriate).  No need to
   recompile the kernel to change these parameters. 

 - Reboot with the new kernel and enjoy. 

IF SOMETHING GOES WRONG:

 - If you have problems that seem to be due to kernel bugs, please check
   the file MAINTAINERS to see if there is a particular person associated
   with the part of the kernel that you are having trouble with. If there
   isn't anyone listed there, then the second best thing is to mail
   them to me ([email protected]), and possibly to any other
   relevant mailing-list or to the newsgroup.

 - In all bug-reports, *please* tell what kernel you are talking about,
   how to duplicate the problem, and what your setup is (use your common
   sense).  If the problem is new, tell me so, and if the problem is
   old, please try to tell me when you first noticed it.

 - If the bug results in a message like

     unable to handle kernel paging request at address C0000010
     Oops: 0002
     EIP:   0010:XXXXXXXX
     eax: xxxxxxxx   ebx: xxxxxxxx   ecx: xxxxxxxx   edx: xxxxxxxx
     esi: xxxxxxxx   edi: xxxxxxxx   ebp: xxxxxxxx
     ds: xxxx  es: xxxx  fs: xxxx  gs: xxxx
     Pid: xx, process nr: xx
     xx xx xx xx xx xx xx xx xx xx

   or similar kernel debugging information on your screen or in your
   system log, please duplicate it *exactly*.  The dump may look
   incomprehensible to you, but it does contain information that may
   help debugging the problem.  The text above the dump is also
   important: it tells something about why the kernel dumped code (in
   the above example, it's due to a bad kernel pointer). More information
   on making sense of the dump is in Documentation/oops-tracing.txt

 - If you compiled the kernel with CONFIG_KALLSYMS you can send the dump
   as is, otherwise you will have to use the "ksymoops" program to make
   sense of the dump (but compiling with CONFIG_KALLSYMS is usually preferred).
   This utility can be downloaded from
   ftp://ftp.<country>.kernel.org/pub/linux/utils/kernel/ksymoops/ .
   Alternatively, you can do the dump lookup by hand:

 - In debugging dumps like the above, it helps enormously if you can
   look up what the EIP value means.  The hex value as such doesn't help
   me or anybody else very much: it will depend on your particular
   kernel setup.  What you should do is take the hex value from the EIP
   line (ignore the "0010:"), and look it up in the kernel namelist to
   see which kernel function contains the offending address.

   To find out the kernel function name, you'll need to find the system
   binary associated with the kernel that exhibited the symptom.  This is
   the file 'linux/vmlinux'.  To extract the namelist and match it against
   the EIP from the kernel crash, do:

     nm vmlinux | sort | less

   This will give you a list of kernel addresses sorted in ascending
   order, from which it is simple to find the function that contains the
   offending address.  Note that the address given by the kernel
   debugging messages will not necessarily match exactly with the
   function addresses (in fact, that is very unlikely), so you can't
   just 'grep' the list: the list will, however, give you the starting
   point of each kernel function, so by looking for the function that
   has a starting address lower than the one you are searching for but
   is followed by a function with a higher address you will find the one
   you want.  In fact, it may be a good idea to include a bit of
   "context" in your problem report, giving a few lines around the
   interesting one. 

   If you for some reason cannot do the above (you have a pre-compiled
   kernel image or similar), telling me as much about your setup as
   possible will help.  Please read the REPORTING-BUGS document for details.

 - Alternatively, you can use gdb on a running kernel. (read-only; i.e. you
   cannot change values or set break points.) To do this, first compile the
   kernel with -g; edit arch/i386/Makefile appropriately, then do a "make
   clean". You'll also need to enable CONFIG_PROC_FS (via "make config").

   After you've rebooted with the new kernel, do "gdb vmlinux /proc/kcore".
   You can now use all the usual gdb commands. The command to look up the
   point where your system crashed is "l *0xXXXXXXXX". (Replace the XXXes
   with the EIP value.)

   gdb'ing a non-running kernel currently fails because gdb (wrongly)
   disregards the starting offset for which the kernel is compiled.

rxe-dev's People

Stargazers

Watchers

rxe-dev's Issues

Remove all RXE_MAX_* module parameters

Need to remove all the RXE_MAX_* module parameter instead use the default values that defined rxe_defice_param enum.

Merge ib_rxe_net with ib_rxe

Need to merge ib_rxe_net kernel module with ib_rxe kernel module so RXE driver will be just one kernel module called ib_rxe.

Complete UD send requests are not sent out due to that QP closed before sending the pkts

Completed UD send requests are not really sent out if the UD QP is closed before RXE sends the packets this cases UD apps to freeze

ib device rex0 not found even though i load the rdma_rxe , and the sh:echo: I/O error

hi,i followed the step to finish configuration of SoftRoCE , but i met some problems i can't fix.

the problem abt I/O error?
btl@ubuntu:~$ sudo rxe_cfg start
sh: echo: I/O error
sh: echo: I/O error
sh: echo: I/O error
sh: echo: I/O error
Use of uninitialized value $rmtu in string at /bin/rxe_cfg line 371.
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
eth0 yes e1000 1500 114.212.22.35 rxe0

2.i can't add device because the I/O error
btl@ubuntu:~$ sudo rxe_cfg add enp5s0
sh: echo: I/O error

3.i have rxe0 device, but it says IB device rxe0 not found
btl@ubuntu:$ sudo rxe_cfg status
Use of uninitialized value $rmtu in string at /bin/rxe_cfg line 371.
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
eth0 yes e1000 1500 114.212.22.35 rxe0
btl@ubuntu:$ ibv_devices
libibverbs: Warning: couldn't load driver 'rxe': librxe-rdmav2.so: cannot open shared object file: No such file or directory
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
device node GUID
------ ----------------

here are some info abt my configuration
btl@ubuntu:$ ls /sys/class/infiniband
rxe0
btl@ubuntu:$ lsmod | grep rxe
rdma_rxe 102400 0
ip6_udp_tunnel 16384 1 rdma_rxe
udp_tunnel 16384 1 rdma_rxe
ib_core 208896 6 rdma_cm,ib_cm,iw_cm,ib_uverbs,rdma_rxe,rdma_ucm
btl@ubuntu:~$ uname -r
4.7.0-rc3+

i will wait for some replies, thank u ~

Remove rxe_bypass_arbiter module parameter

Need to remove rxe_bypass_arbiter module parameter which used to bypass the arbiter task.

Bad memory accounting that results in uncontrolled RXE reserved memory

Need to fix per qp accounting by doing the following:

Move to per-byte accounting instead of number of SKBs.
Remove rxe_max_per_qp module parameter instead define value e.g. 64K
Rewrite remarks - this is a memory management feature, not end-to-end congestion.

Soft lockup on lost packets

I tried to check RXE stability by running perftests for a while on setup that include 3 machine, two machines as clients and one as server.

during the run i saw the following soft lockup:

To reproduce use perftest_2.4:
Server: ib_write_bw -x 0 -p 18516 -D 300 -F
Client: ib_write_bw -x 0 -p 18516 -D 300 -F server-ip

[ 7353.932616] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [ksoftirqd/6:39]
[ 7353.940973] Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver ib_rxe_net ib_rxe rdma_ucm ib_uverbs rdma_cm iw_cm ib_cm ib_umad ib_sa ib_mad ib_core ib_addr nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_CHECKSUM iptable_mangle ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables ipmi_devintf vport_vxlan bridge openvswitch stp llc libcrc32c dm_multipath dm_mod mgag200 ttm nfsd i2c_algo_bit drm_kms_helper auth_rpcgss drm x86_pkg_temp_thermal coretemp nfs_acl kvm_intel kvm i2c_core nfs syscopyarea sysfillrect sysimgblt ioatdma crc32_pclmul ghash_clmulni_intel hpwdt hpilo microcode serio_raw sb_edac edac_core lpc_ich dca mfd_core ipmi_si ipmi_msghandler acpi_power_meter wmi lockd grace sunrpc fscache fuse lp parport mlx4_en vxlan ip6_udp_tunnel udp_tunnel ata_generic pata_acpi sd_mod mlx4_core tg3 ata_piix libata ptp crc32c_intel hpsa pps_core [last unloaded: mlx4_ib]
[ 7353.941039] CPU: 6 PID: 39 Comm: ksoftirqd/6 Not tainted 3.18.0-rc6-rxe-rxe-perf-2015-03-08_15-17-38 #1
[ 7353.941041] Hardware name: HP ProLiant DL380p Gen8, BIOS P70 03/01/2013
[ 7353.941043] task: ffff880829dfd0a0 ti: ffff880829490000 task.ti: ffff880829490000
[ 7353.941045] RIP: 0010:[] [] rxe_responder+0x77/0xb00 [ib_rxe]
[ 7353.941054] RSP: 0018:ffff880829493a48 EFLAGS: 00000246
[ 7353.941055] RAX: 0000000000000000 RBX: ffffffffa05ed7f8 RCX: 0000000000000003
[ 7353.941056] RDX: 0000000000000003 RSI: 0000000000000200 RDI: 0000000000000001
[ 7353.941058] RBP: ffff880829493aa8 R08: 000000000000ffff R09: 000000000000ffff
[ 7353.941059] R10: 0000000000000000 R11: ffff88081d361800 R12: ffff88081d361800
[ 7353.941060] R13: ffff880427a2d180 R14: ffffffff815a8b5c R15: ffff8808294939c8
[ 7353.941062] FS: 0000000000000000(0000) GS:ffff88083f800000(0000) knlGS:0000000000000000
[ 7353.941064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7353.941065] CR2: 00007f3798245d20 CR3: 0000000001a14000 CR4: 00000000000407e0
[ 7353.941066] Stack:
[ 7353.941067] 0000000000000000 0000000000001689 ffff880829493a98 ffffffff815a7f9b
[ 7353.941070] ffff88082959c000 ffff8804164e0a90 ffff880829493a88 ffff8804164e0e18
[ 7353.941072] ffff8804164e0e4c ffff88081d361800 ffff88082959c000 0000000000001689
[ 7353.941074] Call Trace:
[ 7353.941083] [] ? __kmalloc_reserve+0x3b/0xa0
[ 7353.941088] [] rxe_do_task+0x48/0x110 [ib_rxe]
[ 7353.941092] [] rxe_run_task+0x2d/0x34 [ib_rxe]
[ 7353.941095] [] rxe_resp_queue_pkt+0x5f/0x70 [ib_rxe]
[ 7353.941098] [] rxe_rcv+0x1f0/0x210 [ib_rxe]
[ 7353.941102] [] rxe_net_rcv+0xc5/0x140 [ib_rxe_net]
[ 7353.941107] [] __netif_receive_skb_core+0x513/0x650
[ 7353.941110] [] __netif_receive_skb+0x22/0x70
[ 7353.941112] [] netif_receive_skb_internal+0x33/0xa0
[ 7353.941116] [] napi_gro_frags+0xf8/0x120
[ 7353.941122] [] mlx4_en_process_rx_cq+0x696/0x960 [mlx4_en]
[ 7353.941127] [] mlx4_en_poll_rx_cq+0xae/0x170 [mlx4_en]
[ 7353.941130] [] net_rx_action+0x135/0x340
[ 7353.941136] [] __do_softirq+0xf3/0x2c0
[ 7353.941139] [] run_ksoftirqd+0x38/0x50
[ 7353.941144] [] smpboot_thread_fn+0xfd/0x190
[ 7353.941147] [] ? smpboot_create_threads+0x80/0x80
[ 7353.941150] [] kthread+0xce/0xf0
[ 7353.941152] [] ? kthread_freezable_should_stop+0x70/0x70
[ 7353.941159] [] ret_from_fork+0x7c/0xb0
[ 7353.941161] [] ? kthread_freezable_should_stop+0x70/0x70
[ 7353.941162] Code: 4c 8d a3 e4 06 00 00 48 89 55 c8 83 f8 01 45 19 ff 45 31 f6 41 83 e7 1a 41 83 c7 01 44 89 f9 0f 1f 44 00 00 f6 05 7b 13 01 00 04 <0f> 85 3e 0a 00 00 41 83 ff 1d 76 1d be 53 05 00 00 48 c7 c7 c0

Improve Repository Organization

Need to add Readme and possibly Instructions to repository.
Consider including the linux kernel source tree only by reference; this would make it more obvious where the rxe code is.

ibv_devices shows no device even though module ib_rxe has been loaded.

xenmaster1@xenmaster1-OptiPlex-9020:/usr/src/librxe-dev$ rxe_cfg
Use of uninitialized value $rmtu in string at /bin/rxe_cfg line 371.
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
eth0 yes e1000e 1500 10.103.72.14 rxe0
xenmaster1@xenmaster1-OptiPlex-9020:/usr/src/librxe-dev$ ibv_devices
libibverbs: Warning: couldn't load driver 'rxe': librxe-rdmav2.so: cannot open shared object file: No such file or directory
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
device node GUID
------ ----------------

rxe_cfg start and add eth0 has been executed. and lsmod | grep rxe shows that driver has been loaded.

iscsi logout via iSER cause kernel panic

iscsi logout via iSER cause kernel panic, To reproduce need to do the following:

Load ib_rxe driver using rxe_cfg script
Configure RXE device using rxe_cfg script.

Server Side (target):
1- Start target: # tgtd
2- Setup target device: # tgt-setup-lun -n tgt0 -d /tmp/null -b null -t iser

Output from Server:
[root@reg-l-vrt-036-005 ~]# tgtd
[root@reg-l-vrt-036-005 ~]# tgt-setup-lun -n tgt0 -d /tmp/null -b null -t iser
Using transport: iser
Creating new target (name=iqn.2001-04.com.reg-l-vrt-036-005-tgt0, tid=1)
Adding a logical unit (/tmp/null) to target, tid=1
Setting backing store type: null
Accepting connections from all initiators

Client Side (initator):
1- Check if targe is up before loging: # iscsiadm -m node
2- Login to target: # iscsiadm -m node -l
3- Logout from target: # iscsiadm -m node -u

Output from Client:
[root@reg-l-vrt-037-005 ~]# iscsiadm -m node
11.135.196.5:3260,1 iqn.2001-04.com.reg-l-vrt-036-005-tgt0
[root@reg-l-vrt-037-005 ~]# iscsiadm -m node -l
Logging in to [iface: default, target: iqn.2001-04.com.reg-l-vrt-036-005-tgt0, portal:         11.135.196.5,3260] (multiple)
Login to [iface: default, target: iqn.2001-04.com.reg-l-vrt-036-005-tgt0, portal: 11.135.196.5,3260] successful.
[root@reg-l-vrt-037-005 ~]# iscsiadm -m node -u
Logging out of session [sid: 1, target: iqn.2001-04.com.reg-l-vrt-036-005-tgt0, portal: 11.135.196.5,3260]

Kernel Panic:

[14904.748106] iser: iser_conn_terminate: conn ffff8807c021a000 failed to post beacon
[14904.750248] BUG: unable to handle kernel NULL pointer dereference at 0000000000000070
[14904.751136] IP: [] iser_cq_tasklet_fn+0x6a/0x230 [ib_iser]
[14904.751136] PGD 0
[14904.751136] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
[14904.751136] Modules linked in: sd_mod ib_iser libiscsi scsi_transport_iscsi ib_rxe rdma_ucm rdma_cm iw_cm ib_cm ib_uverbs ib_sa ib_mad ib_core mlx4_en ib_addr mlx4_core vxlan ip6_udp_tunnel udp_tunnel ptp pps_core nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ppdev cfg80211 rfkill sg snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm kvm_intel kvm snd_timer snd serio_raw virtio_balloon pcspkr soundcore i2c_piix4 parport_pc parport dm_mirror acpi_cpufreq dm_region_hash dm_log dm_mod nfsd auth_rpcgss nfs_acl lockd grace sunrpc uinput ext4 mbcache jbd2 cirrus ata_generic pata_acpi syscopyarea sysfillrect sysimgblt drm_kms_helper virtio_net ttm virtio_blk drm ata_piix libata i2c_core virtio_pci virtio_ring virtio floppy [last unloaded: mlx4_ib]
[14904.751136] CPU: 2 PID: 17142 Comm: iscsid Not tainted 4.0.0-rc4+ #13
[14904.751136] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[14904.751136] task: ffff8807e02d3460 ti: ffff8807e2814000 task.ti: ffff8807e2814000
[14904.751136] RIP: 0010:[] [] iser_cq_tasklet_fn+0x6a/0x230 [ib_iser]
[14904.751136] RSP: 0018:ffff8807ec003e78 EFLAGS: 00010206
[14904.751136] RAX: 0000000000000000 RBX: ffff8807cf092410 RCX: 0000000000000005
[14904.751136] RDX: 00000000000000b3 RSI: 0000000000000010 RDI: ffff8807d108d100
[14904.751136] RBP: ffff8807ec003ec8 R08: ffff8807d108d100 R09: ffff8807cf6e3000
[14904.751136] R10: ffff8807cf092410 R11: 0000000000002c80 R12: 0000000000000010
[14904.751136] R13: 0000000000000000 R14: ffff8807cf6e3040 R15: ffff8807cf092010
[14904.751136] FS: 00007f33f7659740(0000) GS:ffff8807ec000000(0000) knlGS:0000000000000000
[14904.751136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[14904.751136] CR2: 0000000000000070 CR3: 00000007cf6bc000 CR4: 00000000000007e0
[14904.751136] Stack:
[14904.751136] ffff8807e02d3460 ffffffff8108c99e ffff8807cf092050 ffff8807cf092010
[14904.751136] ffff8807ec003eb8 ffff8807cf092410 0000000000000000 ffff8807cf092418
[14904.751136] 0000000000000100 0000000000000048 ffff8807ec003ef8 ffffffff8108cb97
[14904.751136] Call Trace:
[14904.751136]
[14904.751136] [] ? tasklet_action+0x4e/0x2d0
[14904.751136] [] tasklet_action+0x247/0x2d0
[14904.751136] [] __do_softirq+0xf6/0x670
[14904.751136] [] irq_exit+0x125/0x130
[14904.751136] [] smp_apic_timer_interrupt+0x4a/0x60
[14904.751136] [] apic_timer_interrupt+0x72/0x80
[14904.751136]
[14904.751136] [] ? queue_work_on+0x5b/0xa0
[14904.751136] [] iscsi_iser_ep_disconnect+0xb1/0x120 [ib_iser]
[14904.751136] [] iscsi_if_ep_disconnect.isra.14+0x5f/0x80 [scsi_transport_iscsi]
[14904.751136] [] iscsi_if_recv_msg+0x1136/0x14a0 [scsi_transport_iscsi]
[14904.751136] [] ? iscsi_if_rx+0x3a/0x200 [scsi_transport_iscsi]
[14904.751136] [] ? sched_clock_cpu+0xa8/0xd0
[14904.751136] [] ? iscsi_if_rx+0x3a/0x200 [scsi_transport_iscsi]
[14904.751136] [] ? local_clock+0x15/0x30
[14904.751136] [] ? netlink_deliver_tap+0x87/0x270
[14904.751136] [] iscsi_if_rx+0x96/0x200 [scsi_transport_iscsi]
[14904.751136] [] netlink_unicast+0x103/0x1e0
[14904.751136] [] netlink_sendmsg+0x43d/0x670
[14904.751136] [] ? might_fault+0x5f/0xb0
[14904.751136] [] sock_sendmsg+0x37/0x50
[14904.751136] [] ___sys_sendmsg+0x319/0x330
[14904.751136] [] ? sched_clock+0x9/0x10
[14904.751136] [] ? kvm_clock_read+0x25/0x30
[14904.751136] [] ? sched_clock+0x9/0x10
[14904.751136] [] ? sched_clock_local+0x25/0x90
[14904.751136] [] ? sched_clock_cpu+0xa8/0xd0
[14904.751136] [] ? rcu_read_lock_held+0x6e/0x80
[14904.751136] [] ? __fget_light+0xbe/0xe0
[14904.751136] [] __sys_sendmsg+0x51/0x90
[14904.751136] [] SyS_sendmsg+0x12/0x20
[14904.751136] [] system_call_fastpath+0x12/0x17
[14904.751136] Code: 4c 89 fa ff 90 10 03 00 00 85 c0 41 89 c4 7e 55 8d 58 ff 48 c1 e3 06 48 03 5d c0 66 0f 1f 44 00 00 41 8b 4f 08 49 8b 47 18 85 c9 <4c> 8b 50 70 75 79 41 8b 57 0c 81 fa 80 00 00 00 74 5c 85 d2 0f
[14904.751136] RIP [] iser_cq_tasklet_fn+0x6a/0x230 [ib_iser]
[14904.751136] RSP
[14904.751136] CR2: 0000000000000070
[14904.751136] ------------[ cut here ]------------
[14904.751136] kernel BUG at mm/vmalloc.c:1315!
[14904.751136] invalid opcode: 0000 [#2] SMP DEBUG_PAGEALLOC
[14904.751136] Modules linked in: sd_mod ib_iser libiscsi scsi_transport_iscsi ib_rxe rdma_ucm rdma_cm iw_cm ib_cm ib_uverbs ib_sa ib_mad ib_core mlx4_en ib_addr mlx4_core vxlan ip6_udp_tunnel udp_tunnel ptp pps_core nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ppdev cfg80211 rfkill sg snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm kvm_intel kvm snd_timer snd serio_raw virtio_balloon pcspkr soundcore i2c_piix4 parport_pc parport dm_mirror acpi_cpufreq dm_region_hash dm_log dm_mod nfsd auth_rpcgss nfs_acl lockd grace sunrpc uinput ext4 mbcache jbd2 cirrus ata_generic pata_acpi syscopyarea sysfillrect sysimgblt drm_kms_helper virtio_net ttm virtio_blk drm ata_piix libata i2c_core virtio_pci virtio_ring virtio floppy [last unloaded: mlx4_ib]
[14904.751136] CPU: 2 PID: 17142 Comm: iscsid Not tainted 4.0.0-rc4+ #13
[14904.751136] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[14904.751136] task: ffff8807e02d3460 ti: ffff8807e2814000 task.ti: ffff8807e2814000
[14904.751136] RIP: 0010:[] [] __get_vm_area_node+0x158/0x160
[14904.751136] RSP: 0018:ffff8807ec0032f8 EFLAGS: 00010006
[14904.751136] RAX: 0000000000000100 RBX: 00000000ffffffff RCX: ffffc90000000000
[14904.751136] RDX: 0000000000000022 RSI: 0000000000000001 RDI: 0000000000002000
[14904.751136] RBP: ffff8807ec003338 R08: ffffe8ffffffffff R09: 00000000ffffffff
[14904.751136] R10: ffffc90000000000 R11: 0000000000002cdf R12: 00000000000080d2
[14904.751136] R13: 0000000000001200 R14: 0000000000000080 R15: ffffea0002ea3040
[14904.751136] FS: 00007f33f7659740(0000) GS:ffff8807ec000000(0000) knlGS:0000000000000000
[14904.751136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[14904.751136] CR2: 0000000000000070 CR3: 00000007cf6bc000 CR4: 00000000000007e0
[14904.751136] Stack:
[14904.751136] ffff8807e516ca57 ffffc90000000000 ffff8807ec003438 00000000ffffffff
[14904.751136] 00000000000080d2 0000000000001200 0000000000000080 ffffea0002ea3040
[14904.751136] ffff8807ec0033a8 ffffffff8120a8c3 00000000000080d2 ffffffffa012931d
[14904.751136] Call Trace:
[14904.751136]
[14904.751136] [] __vmalloc_node_range+0x73/0x2c0
[14904.751136] [] ? ttm_tt_init+0x6d/0xb0 [ttm]
[14904.751136] [] ? __probe_kernel_read+0x31/0x60
[14904.751136] [] __vmalloc+0x4a/0x50
[14904.751136] [] ? ttm_tt_init+0x6d/0xb0 [ttm]
[14904.751136] [] ttm_tt_init+0x6d/0xb0 [ttm]
[14904.751136] [] cirrus_ttm_tt_create+0x5c/0x90 [cirrus]
[14904.751136] [] ttm_bo_add_ttm+0x9d/0xe0 [ttm]
[14904.751136] [] ttm_bo_handle_move_mem+0x561/0x5a0 [ttm]
[14904.751136] [] ? leave_mm+0x180/0x180
[14904.751136] [] ? ttm_bo_mem_space+0xe4/0x350 [ttm]
[14904.751136] [] ttm_bo_validate+0x290/0x2b0 [ttm]
[14904.751136] [] ? iounmap+0x84/0xb0
[14904.751136] [] cirrus_bo_push_sysram+0x93/0xe0 [cirrus]
[14904.751136] [] cirrus_crtc_do_set_base.isra.9.constprop.11+0x86/0x420 [cirrus]
[14904.751136] [] cirrus_crtc_mode_set+0x45b/0x4e0 [cirrus]
[14904.751136] [] drm_crtc_helper_set_mode+0x305/0x560 [drm_kms_helper]
[14904.751136] [] drm_crtc_helper_set_config+0x88d/0xab0 [drm_kms_helper]
[14904.751136] [] drm_mode_set_config_internal+0x72/0x120 [drm]
[14904.751136] [] drm_fb_helper_pan_display+0xa0/0x100 [drm_kms_helper]
[14904.751136] [] fb_pan_display+0xda/0x1b0
[14904.751136] [] bit_update_start+0x20/0x50
[14904.751136] [] fbcon_switch+0x3ab/0x5b0
[14904.751136] [] redraw_screen+0x1a9/0x250
[14904.751136] [] fbcon_blank+0x22a/0x2f0
[14904.751136] [] ? mod_timer+0x10e/0x3a0
[14904.751136] [] ? trace_hardirqs_off+0xd/0x10
[14904.751136] [] ? _raw_spin_unlock_irqrestore+0x5f/0x70
[14904.751136] [] ? mod_timer+0x137/0x3a0
[14904.751136] [] do_unblank_screen+0xc0/0x200
[14904.751136] [] unblank_screen+0x10/0x20
[14904.751136] [] bust_spinlocks+0x19/0x40
[14904.751136] [] oops_end+0x3c/0x120
[14904.751136] [] no_context+0x2f3/0x370
[14904.751136] [] __bad_area_nosemaphore+0x7e/0x1d7
[14904.751136] [] ? task_cputime+0x44/0x80
[14904.751136] [] bad_area_nosemaphore+0x13/0x15
[14904.751136] [] __do_page_fault+0x86/0x470
[14904.751136] [] ? __lock_is_held+0x5f/0x90
[14904.751136] [] trace_do_page_fault+0x70/0x440
[14904.751136] [] ? trace_hardirqs_off_thunk+0x3a/0x3c
[14904.751136] [] do_async_page_fault+0x1e/0xd0
[14904.751136] [] async_page_fault+0x28/0x30
[14904.751136] [] ? iser_cq_tasklet_fn+0x6a/0x230 [ib_iser]
[14904.751136] [] ? iser_cq_tasklet_fn+0x48/0x230 [ib_iser]
[14904.751136] [] ? tasklet_action+0x4e/0x2d0
[14904.751136] [] tasklet_action+0x247/0x2d0
[14904.751136] [] __do_softirq+0xf6/0x670
[14904.751136] [] irq_exit+0x125/0x130
[14904.751136] [] smp_apic_timer_interrupt+0x4a/0x60
[14904.751136] [] apic_timer_interrupt+0x72/0x80
[14904.751136]
[14904.751136] [] ? queue_work_on+0x5b/0xa0
[14904.751136] [] iscsi_iser_ep_disconnect+0xb1/0x120 [ib_iser]
[14904.751136] [] iscsi_if_ep_disconnect.isra.14+0x5f/0x80 [scsi_transport_iscsi]
[14904.751136] [] iscsi_if_recv_msg+0x1136/0x14a0 [scsi_transport_iscsi]
[14904.751136] [] ? iscsi_if_rx+0x3a/0x200 [scsi_transport_iscsi]
[14904.751136] [] ? sched_clock_cpu+0xa8/0xd0
[14904.751136] [] ? iscsi_if_rx+0x3a/0x200 [scsi_transport_iscsi]
[14904.751136] [] ? local_clock+0x15/0x30
[14904.751136] [] ? netlink_deliver_tap+0x87/0x270
[14904.751136] [] iscsi_if_rx+0x96/0x200 [scsi_transport_iscsi]
[14904.751136] [] netlink_unicast+0x103/0x1e0
[14904.751136] [] netlink_sendmsg+0x43d/0x670
[14904.751136] [] ? might_fault+0x5f/0xb0
[14904.751136] [] sock_sendmsg+0x37/0x50
[14904.751136] [] ___sys_sendmsg+0x319/0x330
[14904.751136] [] ? sched_clock+0x9/0x10
[14904.751136] [] ? kvm_clock_read+0x25/0x30
[14904.751136] [] ? sched_clock+0x9/0x10
[14904.751136] [] ? sched_clock_local+0x25/0x90
[14904.751136] [] ? sched_clock_cpu+0xa8/0xd0
[14904.751136] [] ? rcu_read_lock_held+0x6e/0x80
[14904.751136] [] ? __fget_light+0xbe/0xe0
[14904.751136] [] __sys_sendmsg+0x51/0x90
[14904.751136] [] SyS_sendmsg+0x12/0x20
[14904.751136] [] system_call_fastpath+0x12/0x17
[14904.751136] Code: 00 00 00 0f bd cf 83 c1 01 83 f9 0c 0f 4c c8 b0 13 83 f9 13 0f 4f c8 49 d3 e6 e9 f5 fe ff ff 48 89 df e8 fc d7 01 00 31 c0 eb b3 <0f> 0b 66 0f 1f 44 00 00 0f 1f 44 00 00 55 49 89 c8 41 b9 ff ff
[14904.751136] RIP [] __get_vm_area_node+0x158/0x160
[14904.751136] RSP
[14904.751136] ---[ end trace 5bfb3e658c1ce2e3 ]---
[14904.751136] Kernel panic - not syncing: Fatal exception in interrupt
[14904.751136] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff)
[14904.751136] drm_kms_helper: panic occurred, switching back to text console
[14904.751136] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
[14905.359695] ------------[ cut here ]------------
[14905.360690] WARNING: CPU: 2 PID: 17142 at arch/x86/kernel/smp.c:124 native_smp_send_reschedule+0x5d/0x60()
[14905.360690] Modules linked in: sd_mod ib_iser libiscsi scsi_transport_iscsi ib_rxe rdma_ucm rdma_cm iw_cm ib_cm ib_uverbs ib_sa ib_mad ib_core mlx4_en ib_addr mlx4_core vxlan ip6_udp_tunnel udp_tunnel ptp pps_core nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ppdev cfg80211 rfkill sg snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm kvm_intel kvm snd_timer snd serio_raw virtio_balloon pcspkr soundcore i2c_piix4 parport_pc parport dm_mirror acpi_cpufreq dm_region_hash dm_log dm_mod nfsd auth_rpcgss nfs_acl lockd grace sunrpc uinput ext4 mbcache jbd2 cirrus ata_generic pata_acpi syscopyarea sysfillrect sysimgblt drm_kms_helper virtio_net ttm virtio_blk drm ata_piix libata i2c_core virtio_pci virtio_ring virtio floppy [last unloaded: mlx4_ib]
[14905.360690] CPU: 2 PID: 17142 Comm: iscsid Tainted: G D 4.0.0-rc4+ #13
[14905.360690] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[14905.360690] 0000000000000000 00000000b1d0bd73 ffff8807ec002d20 ffffffff81743f13
[14905.360690] 0000000000000000 0000000000000000 ffff8807ec002d60 ffffffff81086e1a
[14905.360690] ffff880700000000 0000000000000000 0000000000000002 000000000000e928
[14905.360690] Call Trace:
[14905.360690] [] dump_stack+0x4c/0x65
[14905.360690] [] warn_slowpath_common+0x8a/0xc0
[14905.360690] [] warn_slowpath_null+0x1a/0x20
[14905.360690] [] native_smp_send_reschedule+0x5d/0x60
[14905.360690] [] trigger_load_balance+0x33d/0x470
[14905.360690] [] ? trigger_load_balance+0x74/0x470
[14905.360690] [] scheduler_tick+0x9a/0xe0
[14905.360690] [] update_process_times+0x51/0x60
[14905.360690] [] tick_sched_handle.isra.20+0x25/0x60
[14905.360690] [] tick_sched_timer+0x44/0x80
[14905.360690] [] __run_hrtimer+0xcd/0x6d0
[14905.360690] [] ? hrtimer_interrupt+0x92/0x240
[14905.360690] [] ? tick_sched_do_timer+0x50/0x50
[14905.360690] [] hrtimer_interrupt+0x103/0x240
[14905.360690] [] local_apic_timer_interrupt+0x39/0x60
[14905.360690] [] smp_apic_timer_interrupt+0x45/0x60
[14905.360690] [] apic_timer_interrupt+0x72/0x80
[14905.360690] [] ? panic+0x1c6/0x20e
[14905.360690] [] ? panic+0x1cd/0x20e
[14905.360690] [] ? panic+0x1c6/0x20e
[14905.360690] [] oops_end+0x117/0x120
[14905.360690] [] die+0x4b/0x70
[14905.360690] [] do_trap+0xb0/0x150
[14905.360690] [] do_error_trap+0xa4/0x180
[14905.360690] [] ? sched_clock+0x9/0x10
[14905.360690] [] ? sched_clock_local+0x25/0x90
[14905.360690] [] ? __get_vm_area_node+0x158/0x160
[14905.360690] [] ? select_task_rq_fair+0x4a3/0x7c0
[14905.360690] [] ? select_idle_sibling+0x2b/0x180
[14905.360690] [] ? trace_hardirqs_off_thunk+0x3a/0x3c
[14905.360690] [] do_invalid_op+0x20/0x30
[14905.360690] [] invalid_op+0x1e/0x30
[14905.360690] [] ? __get_vm_area_node+0x158/0x160
[14905.360690] [] ? cirrus_imageblit+0x2f/0x40 [cirrus]
[14905.360690] [] __vmalloc_node_range+0x73/0x2c0
[14905.360690] [] ? ttm_tt_init+0x6d/0xb0 [ttm]
[14905.360690] [] ? __probe_kernel_read+0x31/0x60
[14905.360690] [] __vmalloc+0x4a/0x50
[14905.360690] [] ? ttm_tt_init+0x6d/0xb0 [ttm]
[14905.360690] [] ttm_tt_init+0x6d/0xb0 [ttm]
[14905.360690] [] cirrus_ttm_tt_create+0x5c/0x90 [cirrus]
[14905.360690] [] ttm_bo_add_ttm+0x9d/0xe0 [ttm]
[14905.360690] [] ttm_bo_handle_move_mem+0x561/0x5a0 [ttm]
[14905.360690] [] ? leave_mm+0x180/0x180
[14905.360690] [] ? ttm_bo_mem_space+0xe4/0x350 [ttm]
[14905.360690] [] ttm_bo_validate+0x290/0x2b0 [ttm]
[14905.360690] [] ? iounmap+0x84/0xb0
[14905.360690] [] cirrus_bo_push_sysram+0x93/0xe0 [cirrus]
[14905.360690] [] cirrus_crtc_do_set_base.isra.9.constprop.11+0x86/0x420 [cirrus]
[14905.360690] [] cirrus_crtc_mode_set+0x45b/0x4e0 [cirrus]
[14905.360690] [] drm_crtc_helper_set_mode+0x305/0x560 [drm_kms_helper]
[14905.360690] [] drm_crtc_helper_set_config+0x88d/0xab0 [drm_kms_helper]
[14905.360690] [] drm_mode_set_config_internal+0x72/0x120 [drm]
[14905.360690] [] drm_fb_helper_pan_display+0xa0/0x100 [drm_kms_helper]
[14905.360690] [] fb_pan_display+0xda/0x1b0
[14905.360690] [] bit_update_start+0x20/0x50
[14905.360690] [] fbcon_switch+0x3ab/0x5b0
[14905.360690] [] redraw_screen+0x1a9/0x250
[14905.360690] [] fbcon_blank+0x22a/0x2f0
[14905.360690] [] ? mod_timer+0x10e/0x3a0
[14905.360690] [] ? trace_hardirqs_off+0xd/0x10
[14905.360690] [] ? _raw_spin_unlock_irqrestore+0x5f/0x70
[14905.360690] [] ? mod_timer+0x137/0x3a0
[14905.360690] [] do_unblank_screen+0xc0/0x200
[14905.360690] [] unblank_screen+0x10/0x20
[14905.360690] [] bust_spinlocks+0x19/0x40
[14905.360690] [] oops_end+0x3c/0x120
[14905.360690] [] no_context+0x2f3/0x370
[14905.360690] [] __bad_area_nosemaphore+0x7e/0x1d7
[14905.360690] [] ? task_cputime+0x44/0x80
[14905.360690] [] bad_area_nosemaphore+0x13/0x15
[14905.360690] [] __do_page_fault+0x86/0x470
[14905.360690] [] ? __lock_is_held+0x5f/0x90
[14905.360690] [] trace_do_page_fault+0x70/0x440
[14905.360690] [] ? trace_hardirqs_off_thunk+0x3a/0x3c
[14905.360690] [] do_async_page_fault+0x1e/0xd0
[14905.360690] [] async_page_fault+0x28/0x30
[14905.360690] [] ? iser_cq_tasklet_fn+0x6a/0x230 [ib_iser]
[14905.360690] [] ? iser_cq_tasklet_fn+0x48/0x230 [ib_iser]
[14905.360690] [] ? tasklet_action+0x4e/0x2d0
[14905.360690] [] tasklet_action+0x247/0x2d0
[14905.360690] [] __do_softirq+0xf6/0x670
[14905.360690] [] irq_exit+0x125/0x130
[14905.360690] [] smp_apic_timer_interrupt+0x4a/0x60
[14905.360690] [] apic_timer_interrupt+0x72/0x80
[14905.360690] [] ? queue_work_on+0x5b/0xa0
[14905.360690] [] iscsi_iser_ep_disconnect+0xb1/0x120 [ib_iser]
[14905.360690] [] iscsi_if_ep_disconnect.isra.14+0x5f/0x80 [scsi_transport_iscsi]
[14905.360690] [] iscsi_if_recv_msg+0x1136/0x14a0 [scsi_transport_iscsi]
[14905.360690] [] ? iscsi_if_rx+0x3a/0x200 [scsi_transport_iscsi]
[14905.360690] [] ? sched_clock_cpu+0xa8/0xd0
[14905.360690] [] ? iscsi_if_rx+0x3a/0x200 [scsi_transport_iscsi]
[14905.360690] [] ? local_clock+0x15/0x30
[14905.360690] [] ? netlink_deliver_tap+0x87/0x270
[14905.360690] [] iscsi_if_rx+0x96/0x200 [scsi_transport_iscsi]
[14905.360690] [] netlink_unicast+0x103/0x1e0
[14905.360690] [] netlink_sendmsg+0x43d/0x670
[14905.360690] [] ? might_fault+0x5f/0xb0
[14905.360690] [] sock_sendmsg+0x37/0x50
[14905.360690] [] ___sys_sendmsg+0x319/0x330
[14905.360690] [] ? sched_clock+0x9/0x10
[14905.360690] [] ? kvm_clock_read+0x25/0x30
[14905.360690] [] ? sched_clock+0x9/0x10
[14905.360690] [] ? sched_clock_local+0x25/0x90
[14905.360690] [] ? sched_clock_cpu+0xa8/0xd0
[14905.360690] [] ? rcu_read_lock_held+0x6e/0x80
[14905.360690] [] ? __fget_light+0xbe/0xe0
[14905.360690] [] __sys_sendmsg+0x51/0x90
[14905.360690] [] SyS_sendmsg+0x12/0x20
[14905.360690] [] system_call_fastpath+0x12/0x17
[14905.360690] ---[ end trace 5bfb3e658c1ce2e4 ]---

iscsi loging via iSER cause kernel panic

iscsi loging via iSER cause kernel panic, To reproduce please follow the following steps:

On both sides load ib_rxe kernel module and configure rxe device using rxe_cfg script.
Server (target):

1- start target: # tgtd
2- setup target device: # tgt-setup-lun -n tgt0 -d /tmp/null -b null -t iser

Output from Server:
[root@reg-l-vrt-036-005 ~]# tgtd
[root@reg-l-vrt-036-005 ~]# tgt-setup-lun -n tgt0 -d /tmp/null -b null -t iser
Using transport: iser
Creating new target (name=iqn.2001-04.com.reg-l-vrt-036-005-tgt0, tid=1)
Adding a logical unit (/tmp/null) to target, tid=1
Setting backing store type: null
Accepting connections from all initiators

Client (initiator):

1- check if target is up before login: # iscsiadm -m node
2- login to target: # iscsiadm -m node -l

Output from Client:
[root@reg-l-vrt-037-005 ~]# iscsiadm -m node
11.135.196.5:3260,1 iqn.2001-04.com.reg-l-vrt-036-005-tgt0
[root@reg-l-vrt-037-005 ~]# iscsiadm -m node -l
Logging in to [iface: default, target: iqn.2001-04.com.reg-l-vrt-036-005-tgt0, portal:         11.135.196.5,3260] (multiple)

Kernel Panic:
[ 325.870106] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 325.870854] IP: [] __memcpy+0xd/0x110
[ 325.966869] PGD 0
[ 325.966869] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
[ 325.966869] Modules linked in: ib_iser libiscsi scsi_transport_iscsi ib_rxe rdma_ucm rdma_cm iw_cm ib_cm ib_uverbs ib_sa ib_mad ib_core mlx4_en ib_addr mlx4_core vxlan ip6_udp_tunnel udp_tunnel ptp pps_core nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ppdev cfg80211 rfkill sg snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep kvm_intel snd_seq kvm snd_seq_device snd_pcm serio_raw pcspkr virtio_balloon parport_pc parport snd_timer snd acpi_cpufreq i2c_piix4 soundcore dm_mirror dm_region_hash dm_log dm_mod nfsd auth_rpcgss nfs_acl lockd grace sunrpc uinput ext4 mbcache jbd2 cirrus ata_generic syscopyarea pata_acpi sysfillrect sysimgblt drm_kms_helper virtio_blk virtio_net ttm drm ata_piix libata virtio_pci i2c_core virtio_ring virtio floppy [last unloaded: mlx4_ib]
[ 325.966869] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.0.0-rc4+ #13
[ 325.966869] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[ 325.966869] task: ffff8807e8739a30 ti: ffff8807e8744000 task.ti: ffff8807e8744000
[ 325.966869] RIP: 0010:[] [] __memcpy+0xd/0x110
[ 325.966869] RSP: 0018:ffff8807ec003810 EFLAGS: 00010202
[ 325.966869] RAX: 0000000000000000 RBX: ffff8807e2d572c6 RCX: 0000000000000004
[ 325.966869] RDX: 0000000000000004 RSI: ffff8807e2d572c6 RDI: 0000000000000000
[ 325.966869] RBP: ffff8807ec003898 R08: 0000000000000000 R09: 0000000000000000
[ 325.966869] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
[ 325.966869] R13: 0000000000000024 R14: 0000000000000024 R15: ffff8807e0d0fa28
[ 325.966869] FS: 0000000000000000(0000) GS:ffff8807ec000000(0000) knlGS:0000000000000000
[ 325.966869] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 325.966869] CR2: 0000000000000000 CR3: 0000000001aa2000 CR4: 00000000000007e0
[ 325.966869] Stack:
[ 325.966869] ffffffffa06deb5a ffff8807ce39b0d0 0000000000000004 0000000000000000
[ 325.966869] ffffffffa06d8b16 ffff8807e8739a30 0000000000000000 ffffffff8278fe50
[ 325.966869] ffff8807ce33b8a0 0000000000000411 5076829536caba05 ffff8807ec003898
[ 325.966869] Call Trace:
[ 325.966869]
[ 325.966869] [] ? rxe_mem_copy+0x1ea/0x220 [ib_rxe]
[ 325.966869] [] ? rxe_pool_get_index+0x96/0xc0 [ib_rxe]
[ 325.966869] [] rxe_responder+0x10bc/0x1c60 [ib_rxe]
[ 325.966869] [] ? sched_clock_local+0x25/0x90
[ 325.966869] [] ? local_clock+0x15/0x30
[ 325.966869] [] ? _raw_spin_unlock_irqrestore+0x36/0x70
[ 325.966869] [] rxe_do_task+0x9d/0x110 [ib_rxe]
[ 325.966869] [] rxe_run_task+0x12/0x30 [ib_rxe]
[ 325.966869] [] rxe_resp_queue_pkt+0x50/0x60 [ib_rxe]
[ 325.966869] [] rxe_rcv+0x195/0x3d0 [ib_rxe]
[ 325.966869] [] ? __pskb_pull_tail+0x2a0/0x370
[ 325.966869] [] ? dma_device+0x40/0x40 [ib_rxe]
[ 325.966869] [] rxe_udp_encap_recv+0xa8/0xe0 [ib_rxe]
[ 325.966869] [] udp_queue_rcv_skb+0x309/0x590
[ 325.966869] [] __udp4_lib_rcv+0x4fb/0xad0
[ 325.966869] [] udp_rcv+0x1a/0x20
[ 325.966869] [] ip_local_deliver_finish+0x133/0x3e0
[ 325.966869] [] ? ip_local_deliver_finish+0x4c/0x3e0
[ 325.966869] [] ip_local_deliver+0x48/0x80
[ 325.966869] [] ip_rcv_finish+0x168/0x5c0
[ 325.966869] [] ip_rcv+0x2a2/0x3f0
[ 325.966869] [] __netif_receive_skb_core+0x7e6/0xed0
[ 325.966869] [] ? __netif_receive_skb_core+0x91/0xed0
[ 325.966869] [] __netif_receive_skb+0x18/0x60
[ 325.966869] [] netif_receive_skb_internal+0x40/0x220
[ 325.966869] [] napi_gro_frags+0x249/0x380
[ 325.966869] [] mlx4_en_process_rx_cq+0x76f/0xcc0 [mlx4_en]
[ 325.966869] [] mlx4_en_poll_rx_cq+0x9f/0x170 [mlx4_en]
[ 325.966869] [] net_rx_action+0x16b/0x550
[ 325.966869] [] __do_softirq+0xf6/0x670
[ 325.966869] [] irq_exit+0x125/0x130
[ 325.966869] [] do_IRQ+0x5a/0xf0
[ 325.966869] [] common_interrupt+0x72/0x72
[ 325.966869]
[ 325.966869] [] ? native_safe_halt+0x6/0x10
[ 325.966869] [] ? trace_hardirqs_on+0xd/0x10
[ 325.966869] [] default_idle+0x23/0x240
[ 325.966869] [] arch_cpu_idle+0xf/0x20
[ 325.966869] [] cpu_idle_loop+0x415/0x5e0
[ 325.966869] [] cpu_startup_entry+0x69/0x70
[ 325.966869] [] start_secondary+0x193/0x200
[ 325.966869] Code: 2b 43 50 88 43 4e 48 83 c4 08 5b 5d c3 90 e8 6b fc ff ff eb e6 90 90 90 90 90 90 90 90 90 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 48 a5 89 d1 f3 a4 c3 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c
[ 325.966869] RIP [] __memcpy+0xd/0x110
[ 325.966869] RSP
[ 325.966869] CR2: 0000000000000000
[ 325.966869] ---[ end trace 074e350ea19548c2 ]---
[ 325.966869] Kernel panic - not syncing: Fatal exception in interrupt
[ 325.966869] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000- 0xffffffff9fffffff)
[ 325.966869] drm_kms_helper: panic occurred, switching back to text console
[ 325.966869] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
[ 326.276636] ------------[ cut here ]------------
[ 326.277633] WARNING: CPU: 2 PID: 0 at arch/x86/kernel/smp.c:124 native_smp_send_reschedule+0x5d/0x60()
[ 326.277633] Modules linked in: ib_iser libiscsi scsi_transport_iscsi ib_rxe rdma_ucm rdma_cm iw_cm ib_cm ib_uverbs ib_sa ib_mad ib_core mlx4_en ib_addr mlx4_core vxlan ip6_udp_tunnel udp_tunnel ptp pps_core nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ppdev cfg80211 rfkill sg snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep kvm_intel snd_seq kvm snd_seq_device snd_pcm serio_raw pcspkr virtio_balloon parport_pc parport snd_timer snd acpi_cpufreq i2c_piix4 soundcore dm_mirror dm_region_hash dm_log dm_mod nfsd auth_rpcgss nfs_acl lockd grace sunrpc uinput ext4 mbcache jbd2 cirrus ata_generic syscopyarea pata_acpi sysfillrect sysimgblt drm_kms_helper virtio_blk virtio_net ttm drm ata_piix libata virtio_pci i2c_core virtio_ring virtio floppy [last unloaded: mlx4_ib]
[ 326.277633] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G D 4.0.0-rc4+ #13
[ 326.277633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[ 326.277633] 0000000000000000 5076829536caba05 ffff8807ec003230 ffffffff81743f13
[ 326.277633] 0000000000000000 0000000000000000 ffff8807ec003270 ffffffff81086e1a
[ 326.277633] ffff880700000000 0000000000000000 0000000000000002 000000000000e928
[ 326.277633] Call Trace:
[ 326.277633] [] dump_stack+0x4c/0x65
[ 326.277633] [] warn_slowpath_common+0x8a/0xc0
[ 326.277633] [] warn_slowpath_null+0x1a/0x20
[ 326.277633] [] native_smp_send_reschedule+0x5d/0x60
[ 326.277633] [] trigger_load_balance+0x33d/0x470
[ 326.277633] [] ? trigger_load_balance+0x74/0x470
[ 326.277633] [] scheduler_tick+0x9a/0xe0
[ 326.277633] [] update_process_times+0x51/0x60
[ 326.277633] [] tick_sched_handle.isra.20+0x25/0x60
[ 326.277633] [] tick_sched_timer+0x44/0x80
[ 326.277633] [] __run_hrtimer+0xcd/0x6d0
[ 326.277633] [] ? hrtimer_interrupt+0x92/0x240
[ 326.277633] [] ? tick_sched_do_timer+0x50/0x50
[ 326.277633] [] hrtimer_interrupt+0x103/0x240
[ 326.277633] [] local_apic_timer_interrupt+0x39/0x60
[ 326.277633] [] smp_apic_timer_interrupt+0x45/0x60
[ 326.277633] [] apic_timer_interrupt+0x72/0x80
[ 326.277633] [] ? retint_restore_args+0x13/0x13
[ 326.277633] [] ? panic+0x1c6/0x20e
[ 326.277633] [] ? panic+0x1cd/0x20e
[ 326.277633] [] ? panic+0x1c6/0x20e
[ 326.277633] [] oops_end+0x117/0x120
[ 326.277633] [] no_context+0x2f3/0x370
[ 326.277633] [] __bad_area_nosemaphore+0x7e/0x1d7
[ 326.277633] [] bad_area_nosemaphore+0x13/0x15
[ 326.277633] [] __do_page_fault+0x86/0x470
[ 326.277633] [] ? __lock_is_held+0x5f/0x90
[ 326.277633] [] trace_do_page_fault+0x70/0x440
[ 326.277633] [] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 326.277633] [] do_async_page_fault+0x1e/0xd0
[ 326.277633] [] async_page_fault+0x28/0x30
[ 326.277633] [] ? __memcpy+0xd/0x110
[ 326.277633] [] ? rxe_mem_copy+0x1ea/0x220 [ib_rxe]
[ 326.277633] [] ? rxe_pool_get_index+0x96/0xc0 [ib_rxe]
[ 326.277633] [] rxe_responder+0x10bc/0x1c60 [ib_rxe]
[ 326.277633] [] ? sched_clock_local+0x25/0x90
[ 326.277633] [] ? local_clock+0x15/0x30
[ 326.277633] [] ? _raw_spin_unlock_irqrestore+0x36/0x70
[ 326.277633] [] rxe_do_task+0x9d/0x110 [ib_rxe]
[ 326.277633] [] rxe_run_task+0x12/0x30 [ib_rxe]
[ 326.277633] [] rxe_resp_queue_pkt+0x50/0x60 [ib_rxe]
[ 326.277633] [] rxe_rcv+0x195/0x3d0 [ib_rxe]
[ 326.277633] [] ? __pskb_pull_tail+0x2a0/0x370
[ 326.277633] [] ? dma_device+0x40/0x40 [ib_rxe]
[ 326.277633] [] rxe_udp_encap_recv+0xa8/0xe0 [ib_rxe]
[ 326.277633] [] udp_queue_rcv_skb+0x309/0x590
[ 326.277633] [] __udp4_lib_rcv+0x4fb/0xad0
[ 326.277633] [] udp_rcv+0x1a/0x20
[ 326.277633] [] ip_local_deliver_finish+0x133/0x3e0
[ 326.277633] [] ? ip_local_deliver_finish+0x4c/0x3e0
[ 326.277633] [] ip_local_deliver+0x48/0x80
[ 326.277633] [] ip_rcv_finish+0x168/0x5c0
[ 326.277633] [] ip_rcv+0x2a2/0x3f0
[ 326.277633] [] __netif_receive_skb_core+0x7e6/0xed0
[ 326.277633] [] ? __netif_receive_skb_core+0x91/0xed0
[ 326.277633] [] __netif_receive_skb+0x18/0x60
[ 326.277633] [] netif_receive_skb_internal+0x40/0x220
[ 326.277633] [] napi_gro_frags+0x249/0x380
[ 326.277633] [] mlx4_en_process_rx_cq+0x76f/0xcc0 [mlx4_en]
[ 326.277633] [] mlx4_en_poll_rx_cq+0x9f/0x170 [mlx4_en]
[ 326.277633] [] net_rx_action+0x16b/0x550
[ 326.277633] [] __do_softirq+0xf6/0x670
[ 326.277633] [] irq_exit+0x125/0x130
[ 326.277633] [] do_IRQ+0x5a/0xf0
[ 326.277633] [] common_interrupt+0x72/0x72
[ 326.277633] [] ? native_safe_halt+0x6/0x10
[ 326.277633] [] ? trace_hardirqs_on+0xd/0x10
[ 326.277633] [] default_idle+0x23/0x240
[ 326.277633] [] arch_cpu_idle+0xf/0x20
[ 326.277633] [] cpu_idle_loop+0x415/0x5e0
[ 326.277633] [] cpu_startup_entry+0x69/0x70
[ 326.277633] [] start_secondary+0x193/0x200
[ 326.277633] ---[ end trace 074e350ea19548c3 ]---

ibstatus reports incorrect link rate when 1000BASE-T is used

I am having lots of fun testing Soft-RoCE on my VM. I am running v4.8-rc5 and aside from a few trips along the way things seem to be going quite well. I do notice that the speed reported by ibstatus is not correct for a 10/100/100BASE-T NIC operating in 1000BASE-T mode:

root@donard-qemu:~/librxe-dev# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
MDI-X: off (auto)
Supports Wake-on: umbg
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

root@donard-qemu:~/librxe-dev# ibstatus
Infiniband device 'rxe0' port 1 status:
default gid: fe80:0000:0000:0000:5054:00ff:fe12:3456
base lid: 0x0
sm lid: 0x0
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 2.5 Gb/sec (1X)
link_layer: Ethernet

root@donard-qemu:~/librxe-dev# rxe_cfg status
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
eth0 yes e1000 1500 10.0.2.15 rxe0 1024 (3)

Cheers

Stephen

Fix the check of max unAcked PSNs

Need to fix the check of max unAcked PSNs in the rxe_requester function.

Executing 'rxe_cfg start' gots 'echo: write error: Invalid argument'

Hi,
I have installed rxe-v18 and librxe-dev on centos 6.5

[admin@localhost librxe-dev]$ uname -r
4.7.0-rc3+

When I try to load the ib_rxe kernel module or add Ethernet interface, I get the following errors:

[root@localhost ~]# rxe_cfg start
sh: line 0: echo: write error: Invalid argument
sh: line 0: echo: write error: Invaild argument
  Name   Link  Driver  Speed  NMTU  IPv4_addr    RDEV  RMTU
  p4p2   yes   r8169          1500  192.168.3.2
  wlan0  no    ath9k          1500

[root@localhost ~]# rxe_cfg add eth0
sh: line 0: echo: write error: Invalid argument

It seems that, the following codes in the rxe_cfg failed:

sub rxe_add {
    my $eth = $_[0];

    if (!($eth =~ /[\w]+[\d]/)) {
        print "eth_name ($eth) looks bogus\n";
        return;
    }

    if (!defined($rxe_names{$eth})) {   
        system("echo '$eth' > $parms/add"); ## this line failed
    }
    if (!$no_persist) {
        add_persistent($eth);
        commit_persistent();
    }
}

Is there any solution ?

Regards,
Tao.

Remove fast* module parameters

Need to remove fast* module parameters which used to bypass tasklet.

The MAKE process didn't finished correctly.

When I execute the MAKE -j 32 command，there is an error as follows：
make[1]: *** No rules can create “fs/built-in.o” Require goals：“fs/proc_namespace.o”. Stop。
make: *** [fs] error 2

rxe failed connectivity test

hi @monis410 , I'm a RDMA beginner. I met a problem very similar to the previous issue (#49).

mcfatealan@mcfatealan-desktop:~/librxe-dev$ ibv_devices
libibverbs: Warning: couldn't load driver 'rxe': librxe-rdmav2.so: cannot open shared object file: No such file or directory
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
    device                 node GUID
    ------              ----------------

I walk around it by moving /usr/lib64/* to /usr/lib/. But after that I have problems on connectivity tests.

My OS is Ubuntu 16.04 LTS (4.7.0-rc3+).

Some of my test result:

mcfatealan@mcfatealan-desktop:~/librxe-dev$ sudo rxe_cfg start 
sh: echo: I/O error
sh: echo: I/O error
sh: echo: I/O error
sh: echo: I/O error
  Name    Link  Driver  Speed  NMTU  IPv4_addr      RDEV  RMTU          
  enp5s0  yes   r8169          1500  192.168.10.19  rxe0  1024  (3)  

mcfatealan@mcfatealan-desktop:~/librxe-dev$ lsmod | grep rxe
rdma_rxe              102400  0
ip6_udp_tunnel         16384  1 rdma_rxe
udp_tunnel             16384  1 rdma_rxe
ib_core               208896  6 rdma_cm,ib_cm,iw_cm,ib_uverbs,rdma_rxe,rdma_ucm

mcfatealan@mcfatealan-desktop:~/librxe-dev$ lsmod | grep ib_uverbs
ib_uverbs              61440  1 rdma_ucm
ib_core               208896  6 rdma_cm,ib_cm,iw_cm,ib_uverbs,rdma_rxe,rdma_ucm

mcfatealan@mcfatealan-desktop:~/librxe-dev$ ibv_devices
    device                 node GUID
    ------              ----------------
    rxe0                be5ff4fffe3acd36

mcfatealan@mcfatealan-desktop:~/librxe-dev$ ibv_devinfo -d rxe0
hca_id: rxe0
    transport:          InfiniBand (0)
    fw_ver:             0.0.0
    node_guid:          be5f:f4ff:fe3a:cd36
    sys_image_guid:         0000:0000:0000:0000
    vendor_id:          0x0000
    vendor_part_id:         0
    hw_ver:             0x0
    phys_port_cnt:          1
        port:   1
            state:          PORT_ACTIVE (4)
            max_mtu:        4096 (5)
            active_mtu:     1024 (3)
            sm_lid:         0
            port_lid:       0
            port_lmc:       0x00
            link_layer:     Ethernet

Then I tested connectivity both on one machines(self-to-self), and on one physical machine and a virtual machine. I can ping each other, so the connectivity of these machines is fine. The test result is exactly the same.

server:
mcfatealan@mcfatealan-desktop:~/librxe-dev$ ibv_rc_pingpong -g 0 -d rxe0 -i 1
  local address:  LID 0x0000, QPN 0x000011, PSN 0x2b7bf6, GID fe80::be5f:f4ff:fe3a:cd36
  remote address: LID 0x0000, QPN 0x000012, PSN 0x4255a9, GID fe80::be5f:f4ff:fe3a:cd36
//hanging...

client:
mcfatealan@mcfatealan-desktop:~$ ibv_rc_pingpong -g 0 -d rxe0 -i 1 192.168.10.19
  local address:  LID 0x0000, QPN 0x000012, PSN 0x4255a9, GID fe80::be5f:f4ff:fe3a:cd36
  remote address: LID 0x0000, QPN 0x000011, PSN 0x2b7bf6, GID fe80::be5f:f4ff:fe3a:cd36
//hanging...




server:
mcfatealan@mcfatealan-desktop:~/librxe-dev$ rping -s -a 192.168.10.19 -v -C 10
//hanging...

client:
mcfatealan@mcfatealan-desktop:~/librxe-dev$ rping -c -a 192.168.10.19 -v -C 10
//hanging...

Could you help me have a look on it? Thank you so much!

BTW, could Soft RoCE work well with python-rdma (https://github.com/jgunthorpe/python-rdma)? I tested that too and failed, not sure whether both two problems share the same root.

Kernel warning trace while handle DMA within RXE

I got the following kernel warning after install the kernel with debug flags.

To reproduce:
1- Server: ucmatose
2- Client: ucmatose -s <server_ip_address>
3- Check dmesg for the following warning

[  235.091672] WARNING: CPU: 1 PID: 2873 at lib/dma-debug.c:601     debug_dma_assert_idle+0x198/0x210()
[  235.091674] mlx4_core 0000:00:07.0: DMA-API: cpu touching an active dma mapped cacheline     [cln=0x000000001f405880]
[  235.092834] Modules linked in: ib_rxe rdma_ucm rdma_cm iw_cm ib_cm ib_uverbs ib_sa ib_mad     ib_core mlx4_en ib_addr mlx4_core vxlan ip6_udp_tunnel udp_tunnel ptp pps_core nfsv3     rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache cfg80211 rfkill sg dm_mirror dm_region_hash     dm_log dm_mod ppdev snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec     snd_hda_core snd_hwdep kvm_intel snd_seq kvm snd_seq_device snd_pcm snd_timer     virtio_balloon snd pcspkr serio_raw i2c_piix4 soundcore parport_pc parport acpi_cpufreq nfsd     auth_rpcgss nfs_acl lockd grace sunrpc uinput ext4 mbcache jbd2 ata_generic pata_acpi virtio_blk     virtio_net cirrus syscopyarea sysfillrect sysimgblt drm_kms_helper ttm drm ata_piix libata virtio_pci     virtio_ring i2c_core virtio floppy [last unloaded: mlx4_ib]
[  235.092840] CPU: 1 PID: 2873 Comm: bash Not tainted 4.0.0+ #1
[  235.092841] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[  235.092845]  0000000000000000 00000000576081c5 ffff8807d3b9bb98 ffffffff8174e720
[  235.092847]  0000000000000000 ffff8807d3b9bbf0 ffff8807d3b9bbd8 ffffffff8108ad9a
[  235.092848]  ffff8800badf1a40 ffff8807e2e55a80 ffff8807e82abaf0 00007fc29680ba00
[  235.092849] Call Trace:
[  235.092856]  [<ffffffff8174e720>] dump_stack+0x4c/0x65
[  235.092860]  [<ffffffff8108ad9a>] warn_slowpath_common+0x8a/0xc0
[  235.092863]  [<ffffffff8108ae25>] warn_slowpath_fmt+0x55/0x70
[  235.092866]  [<ffffffff810e921d>] ? trace_hardirqs_on+0xd/0x10
[  235.092869]  [<ffffffff813bd7b8>] debug_dma_assert_idle+0x198/0x210
[  235.092871]  [<ffffffff811fd5e9>] wp_page_copy.isra.47+0xa9/0x4c0
[  235.092872]  [<ffffffff811ff1c6>] ? do_wp_page+0x86/0x500
[  235.092874]  [<ffffffff811ff1df>] do_wp_page+0x9f/0x500
[  235.092875]  [<ffffffff8120103e>] ? handle_mm_fault+0x56e/0x17b0
[  235.092877]  [<ffffffff81201763>] handle_mm_fault+0xc93/0x17b0
[  235.092879]  [<ffffffff810ea6af>] ? __lock_is_held+0x5f/0x90
[  235.092883]  [<ffffffff81074f8e>] __do_page_fault+0x1ae/0x470
[  235.092886]  [<ffffffff81165196>] ? __audit_syscall_exit+0x1e6/0x280
[  235.092888]  [<ffffffff81075342>] trace_do_page_fault+0x72/0x450
[  235.092903]  [<ffffffff8139a3d4>] ? trace_hardirqs_off_thunk+0x17/0x19
[  235.092906]  [<ffffffff8106ee7e>] do_async_page_fault+0x1e/0xe0
[  235.092908]  [<ffffffff8175aa18>] async_page_fault+0x28/0x30
[  235.092910] ---[ end trace 0ebbc9d6a84fcf81 ]---
[  235.092910] Mapped at:
[  235.092912]  [<ffffffff813bb92c>] debug_dma_map_sg+0x12c/0x150
[  235.092937]  [<ffffffffa0672da7>] ib_umem_get+0x457/0x4f0 [ib_core]
[  235.092941]  [<ffffffffa06e8469>] rxe_mem_init_user+0x49/0x270 [ib_rxe]
[  235.092944]  [<ffffffffa06e4b4e>] rxe_reg_user_mr+0x9e/0x130 [ib_rxe]
[  235.092948]  [<ffffffffa0583901>] ib_uverbs_reg_mr+0x171/0x3c0 [ib_uverbs]

Remove disable CRC code and module parameter

Need to remove disable CRC code and module parameter which use to disable crc32 computation on outbound packets, Need to compute CRC for all outbound packets without giving the user the option to choose.

Bad doorbell system call

Bad doorbell system call and improper sharing of kernel structures with use without proper kapi/ definitions.

Netfilter definitions for RoCE

Similar to Issue #10, rxe_net.h contains macro-defined constants for NFPROTO_RXE, commented to be defined in .../include/linux/netfilter.h, and NF_RXE_IN and NF_RXE_OUT, commented to be defined in .../include/linux/netfilter_rxe.h.

Use ib_umem_offset in kernel 3.19

Due to changes in kernel API need to change the way of extracting the offset from umem struct, need to use new function called ib_umem_offset.

Prepare test plan document and upload it to github

Need to prepare test plan document that will describe how to test RXE driver and upload it to github.

install issue

when I do the make for librxe-dev I get:
src/rxe.c:61:25: fatal error: rdma/ib_rxe.h: No such file or directory

include <rdma/ib_rxe.h>

using CentOS 7, 4.0.0

RoCEv2 Support

Need to change RXE code to support RoCEv2

Remove use of tasklets within RXE

Remove the use of tasklets withing RXE kernel module that cause multiple asynch bugs and inefficiencies

Remove manual MTU configuration

Need to remove manual MTU configuration that done via sysfs and respond only to Netdev event, to do this we need to do the following:
1- Remove from rxe_cfg script the option to set MTU option.
2- Remove rxe_param_set_mtu from rxe_net_sysfs.c file.

Create ethertype for RoCE

Need to create new ethertype for RoCE "ETH_P_ROCE"under "include/uapi/linux/if_ether.h" and change the code to use the new ethertype instead of using "ETH_P_RXE" ethertype.

RDMA CM apps are not working over RXE

RDMA CM application are not working over RXE devices.

To reproduce:
rping -s -a 11.195.35.1 -v
rdma_bind_addr: No such file or directory

Destination mac address is zeros for UD traffic

When trying to send UD traffic using ibv_ud_pingpong the application get stuck and in the tcpdump output i see that the destination mac address is zeros.

Remove ib_rxe_sample kernel module

Need to remove ib_rxe_sample kernel module this done by removing rxe_sample.c file and update the Makefile, Kconfig files.

Issue with IPv6 UDP Tunnel when rxe is built into kernel

Not sure if this is a major issue but I did notice than when I build my kernel with the RXE module built-in that it would fail to load at boot (on a VM) because it was failing to setup a IPv6 UDP tunnel. The git diff below resolved my issue for me.

If I build RXE as a module and load it after boot then I don't see an issue so I suspect this is an ordering issue and RXE is trying to setup the tunnel before IPv6 services are up and running?

Stephen

batesste@cgy1-donard:~/kernels/linux$ git diff
diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c
index 0b8d2ea..d92d0b4 100644
--- a/drivers/infiniband/sw/rxe/rxe_net.c
+++ b/drivers/infiniband/sw/rxe/rxe_net.c
@@ -668,13 +668,13 @@ int rxe_net_init(void)

    spin_lock_init(&dev_list_lock);

```
  recv_sockets.sk6 = rxe_setup_udp_tunnel(&init_net,
```
+/* recv_sockets.sk6 = rxe_setup_udp_tunnel(&init_net,
htons(ROCE_V2_UDP_DPORT), true);
if (IS_ERR(recv_sockets.sk6)) {
recv_sockets.sk6 = NULL;
pr_err("rxe: Failed to create IPv6 UDP tunnel\n");
return -1;
```
  }
```

   }*/

recv_sockets.sk4 = rxe_setup_udp_tunnel(&init_net,
                htons(ROCE_V2_UDP_DPORT), false);

kernel Panic

Hi All,

Have two machines with normal NIC adapter on it. On machine acting as NVMe-Host and the other machine NVMe-Target. Target is NULL_BLOCK_DEVICE provided by linux. Discovery/Connect NVMe commands are working fine. Data transfer is happening fine through the Soft-RoCE interface.

When tried running IO's [Read] using fio command, NVMe-Host tries to re-connect to the target and then kernel panic happens. Stack trace shows the error in rdma_disconnect().

Below is the stack trace when panic happened.
Sep 16 16:40:44 john kernel: [ 4660.937003] nvme nvme0: rdma_resolve_addr wait failed (-104).
Sep 16 16:40:53 john kernel: [ 4669.289136] rxe: set rxe0 active
Sep 16 16:40:53 john kernel: [ 4669.289138] rxe: added rxe0 to eno1
Sep 16 16:40:53 john kernel: [ 4669.291500] interface en01 not found
Sep 16 16:41:03 john kernel: [ 4679.172136] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.0.154:1023
Sep 16 16:41:05 john kernel: [ 4681.896008] nvme nvme0: creating 4 I/O queues.
Sep 16 16:41:05 john kernel: [ 4681.928447] nvme nvme0: new ctrl: NQN "testsubsystem", addr 192.168.0.154:1023
[ 5128.118832] blk_update_request: I/O error, dev nvme0n1, sector 664872
[ 5128.125658] blk_update_request: I/O error, dev nvme0n1, sector 1614312
[ 5128.132569] blk_update_request: I/O error, dev nvme0n1, sector 1309672
[ 5128.139307] blk_update_request: I/O error, dev nvme0n1, sector 1240976
Sep 16 16:48:32 [ 5128.146346] blk_update_request: I/O error, dev nvme0n1, sector 2037616
john kernel: [ 5[ 5128.154293] blk_update_request: I/O error, dev nvme0n1, sector 450352
128.118832] blk_[ 5128.162782] blk_update_request: I/O error, dev nvme0n1, sector 1719776
update_request: [ 5128.170989] blk_update_request: I/O error, dev nvme0n1, sector 441656
I/O error, dev n[ 5128.178936] blk_update_request: I/O error, dev nvme0n1, sector 668736
vme0n1, sector 6[ 5128.187821] blk_update_request: I/O error, dev nvme0n1, sector 1249384
64872
Sep 16 16:48:32 john kernel: [ 5128.125658] blk_update_request: I/O error, dev nvme0n1, sector 1614312
Sep 16 16:48:32 john kernel: [ 5128.132569] blk_update_request: I/O error, dev nvme0n1, sector 1309672
Sep 16 16:48:32 john kernel: [ 5128.139307] blk_update_request: I/O error, dev nvme0n1, sector 1240976
Sep 16 16:48:32 john kernel: [ 5128.146346] blk_update_request: I/O error, dev nvme0n1, sector 2037616
Sep 16 16:48:32 john kernel: [ 5128.154293] blk_update_request: I/O error, dev nvme0n1, sector 450352
Sep 16 16:48:32 john kernel: [ 5128.162782] blk_update_request: I/O error, dev nvme0n1, sector 1719776
Sep 16 16:48:32 john kernel: [ 5128.170989] blk_update_request: I/O error, dev nvme0n1, sector 441656
Sep 16 16:48:32 john kernel: [ 5128.178936] blk_update_request: I/O error, dev nvme0n1, sector 668736
Sep 16 16:48:32 john kernel: [ 5128.187821] blk_update_request: I/O error, dev nvme0n1, sector 1249384
Sep 16 16:48:32 john kernel: [ 5128.195526] nvme nvme0: reconnecting in 10 seconds
[ 5149.206030] nvme nvme0: failed nvme_keep_alive_end_io error=16391
Sep 16 16:48:53 john kernel: [ 5149.206030] nvme nvme0: failed nvme_keep_alive_end_io error=16391
[ 5198.356270] nvme nvme0: Connect command failed, error wo/DNR bit: 7
Sep 16 16:49:42 john kernel: [ 5198.356270] nvme nvme0: Connect command failed, error wo/DNR bit: 7
Sep 16 16:49:42 john kernel: [ 5198.362922] nvme nvme0: Failed reconnect attempt, requeueing...
Sep 16 16:49:53 john kernel: [ 5209.619737] nvme nvme0: rdma_resolve_addr wait failed (-110).
Sep 16 16:49:53 john kernel: [ 5209.620031] nvme nvme0: Failed reconnect attempt, requeueing...
[ 5219.859419] general protection fault: 0000 [#1] SMP
[ 5219.864479] Modules linked in: rdma_ucm ib_uverbs nvme_rdma(OE) rdma_cm iw_cm ib_cm configfs nvme_fabrics(OE) nvme_core(OE) rdma_rxe ip6_udp_tunnel udp_tunnel ib_core binfmt_misc snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel intel_powerclamp snd_hda_codec coretemp kvm_intel snd_hda_core kvm snd_hwdep snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq gpio_ich joydev snd_seq_device input_leds snd_timer snd irqbypass mei_me serio_raw mei soundcore lpc_ich mac_hid parport_pc ppdev lp parport autofs4 i915 hid_microsoft hid_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops e1000e psmouse usbhid ptp hid drm pps_core pata_acpi fjes video
[ 5219.931164] CPU: 3 PID: 4130 Comm: kworker/3:0 Tainted: G OE 4.8.0-rc1+ #1
[ 5219.939458] Hardware name: /DH55TC, BIOS TCIBX10H.86A.0037.2010.0614.1712 06/14/2010
[ 5219.949302] Workqueue: nvme_rdma_wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
[ 5219.956929] task: ffff8d0d2b8b4240 task.stack: ffff8d0d87ab8000
[ 5219.963223] RIP: 0010:[] [] rdma_disconnect+0x2e/0x90 [rdma_cm]
[ 5219.972958] RSP: 0018:ffff8d0d87abbdb0 EFLAGS: 00010206
[ 5219.978541] RAX: 6e5f656572745f88 RBX: ffff8d0d34914400 RCX: 0000000000000001
[ 5219.986052] RDX: ffff8d0d34917800 RSI: ffff8d0d35cd8580 RDI: ffff8d0d2b399a00
[ 5219.993504] RBP: ffff8d0d87abbdb8 R08: ffff8d0da34d8c40 R09: 0000000000000002
[ 5220.001116] R10: 0000000000000000 R11: 0000000000003000 R12: ffff8d0d915e9930
[ 5220.008680] R13: ffffe58dffac2600 R14: 00000000000000c0 R15: ffff8d0d915e9930
[ 5220.016211] FS: 0000000000000000(0000) GS:ffff8d0da34c0000(0000) knlGS:0000000000000000
[ 5220.024747] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5220.030719] CR2: 0000556ef4ce1db8 CR3: 00000000afe06000 CR4: 00000000000006e0
[ 5220.038181] Stack:
[ 5220.040308] ffff8d0d914b2400 ffff8d0d87abbdd0 ffffffffc061184e ffff8d0d915e9800
[ 5220.048170] ffff8d0d87abbdf8 ffffffffc0611aef ffff8d0d909b2480 ffff8d0da34d8c40
[ 5220.056159] ffffe58dffac2600 ffff8d0d87abbe38 ffffffff8909eac2 0000000000000000
[ 5220.064048] Call Trace:
[ 5220.066635] [] nvme_rdma_stop_and_free_queue+0x1e/0x40 [nvme_rdma]
[ 5220.074886] [] nvme_rdma_reconnect_ctrl_work+0x7f/0x1d0 [nvme_rdma]
[ 5220.083235] [] process_one_work+0x162/0x4b0
[ 5220.089394] [] worker_thread+0x4b/0x4f0
[ 5220.095199] [] ? process_one_work+0x4b0/0x4b0
[ 5220.101693] [] ? process_one_work+0x4b0/0x4b0
[ 5220.108080] [] kthread+0xf8/0x110
[ 5220.113441] [] ret_from_fork+0x1f/0x40
[ 5220.119170] [] ? kthread_worker_fn+0x1a0/0x1a0
[ 5220.125594] Code: 66 90 55 48 89 e5 53 48 89 fb 48 8b bf 00 03 00 00 48 85 ff 74 65 0f b6 83 b8 01 00 00 48 8b 13 48 c1 e0 04 48 03 82 f8 00 00 00 <8b> 50 08 f6 c2 04 75 14 83 e2 08 b8 ea ff ff ff 74 07 31 f6 e8
[ 5220.146752] RIP [] rdma_disconnect+0x2e/0x90 [rdma_cm]
[ 5220.153918] RSP
[ 5220.168895] ---[ end trace 4e3fbc3ad0b11617 ]---
[ 5220.168899] Kernel panic - not syncing: Fatal exception

Regards
John

Replace net_info array with a linked list

Need to replace the net_info array in rxe_net.h with a linked list and every place that accessing the array change it to accessing the linked list.

After doing this no need for RXE_MAX_IF_INDEX macro

No rdma_rxe

After compiling and installing the kernel,i can't find the moudle of rdma_rxe(rdma_rxe.ko).
I was following the instructions and during the building there was no error .

qipeng@qipeng-OptiPlex-380:/boot/grub$ modinfo rdma_rxe
modinfo: ERROR: Module rdma_rxe not found
but there is ib_rxe:
filename: /lib/modules/4.0.0+/kernel/drivers/infiniband/hw/rxe/ib_rxe.ko

Remove rxe_xmit_shortcut parameter

Need to remove rxe_xmit_shortcut parameter which use to bypass queuing discipline.

IB test fails for ibv_rc_pingpong and ibv_read_bw under RHEL6

I followed instruction from https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home to configure and
install SoftROCE, rping and ibv_write_bw works but no response for ibv_rc_pingpong and ibv_read_bw.

How to troubleshoot this issue?

Environment
Two VMs of CentOS 6.7 (Using rxe 4.0.0+ kernel)
SoftROCE:
rxe-dev: ec91e94
librxe-dev: fd829a95921cfc3f8e1108fcf84d7e8ac64f4bca

rxe_cfg status

Name  Link  Driver  Speed  NMTU  IPv4_addr        RDEV  RMTU
eth0  yes   e1000          1500  192.168.122.219  rxe0  1024  (3)

ibv_rc_pingpong

[root@test]# ibv_rc_pingpong -d rxe0 -g 0 <Peer IP>
  local address:  LID 0x0000, QPN 0x000011, PSN 0x017d96, GID fe80::5054:ff:fe6c:16f1
  remote address: LID 0x0000, QPN 0x000011, PSN 0xd7af63, GID fe80::5054:ff:fead:b03
**--Hang--**

ibv_read_bw

Device not recognized to implement inline feature. Disabling it
.....
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF      Device         : rxe0
 Number of qps   : 1        Transport type : IB
 Connection type : RC       Using SRQ      : OFF
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 Gid index       : 0
 Outstand reads  : 128
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0011 PSN 0xcb231f OUT 0x80 RKey 0x000411 VAddr 0x007f696bb5e000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:122:30
 remote address: LID 0000 QPN 0x0011 PSN 0x71e12d OUT 0x80 RKey 0x000411 VAddr 0x007f83f17ff000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:122:219
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
**--Hang--**

/var/log/messages
test kernel: no qp matches qpn 0x11

Fix number of outstanding SKBs per qp

Need to fix per qp outstanding SKBs by doing the following:
1- Remove the module parameter rxe_max_skb_per_qp.
2- Decrease the number of outstanding SKBs from 800 to 64.
3- Change the comment that tell this is a heuristic algorithm to something like "bounding the number of outstanding SKBs".

Kernel error message while processing QP with SRQs during close flow

When running ibv_srq_pingpong application the following error message appear in the dmesg.

To reproduce:

Run in server side: ibv_srq_pingpong -n 1 -d rxe0 -g 0
Run in client side: ibv_srq_pingpong -n 1 -d rxe0 -g 0 10.135.230.5

From dmesg:
[45295.394016] no qp matches qpn 0x1c
[45295.394021] no qp matches qpn 0x1d
[45295.394023] no qp matches qpn 0x1e
[45295.394025] no qp matches qpn 0x1f

Destroy SRQ does not fail if associated QPs exist

If this is indeed incorrect, I propose adding an integer member to struct rxe_srq to count the number of associated QPs and update the Create QP, Delete QP, and Delete SRQ verbs accordingly.

Here is the IB Spec statement for reference:
o10-10.2.3: If the HCA supports SRQ and the Consumer invokes the Destroy SRQ verb while there are QPs still associated with the SRQ, the CI shall return an Immediate Error.

error while running "rxe_cfg start"

root@gl8:/home/gl8/Desktop/nvme_softroce/librxe-dev# rxe_cfg start
mkdir: cannot create directory `/sys/kernel/config/rdma_cm': Operation not permitted
tee: /sys/kernel/config/rdma_cm/rxe0/default_roce_mode: No such file or directory
IBOE V2
Use of uninitialized value within %eth_driver in string at /bin/rxe_cfg line 361.
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
eth0 1500 rxe0 1024 (3)

Implement backoff timer instead of busy wait

Need to replace the busy wait in the xmit_one_packet function with backoff timer, this will prevent use from overloading the CPU.

Machine crash when working over ipv6 with missing ipv6 address

Stepts to reproduce:

server side:
Hard RoCE: MLNX_OFED_LINUX-3.0-0.2.0
Use ipv6 gid and run ibv_rc_pingpong server:
ibv_rc_pingpong -d mlx4_0 -n 1 -g 0
client side:
Soft RXE
The interface didn't get ipv6 address.
Use ipv6 gid and run ibv_rc_pingpong client:
ibv_rc_pingpong -d rxe0 -n 1 -g 0 10.135.198.6

Client side machine (soft RXE) will crash.

Multicast doesn't work

Need to check multicast traffic after RoCEv2 to see if it's not affected.

RoCEv2 CRC doesn't match HW CRC

Need to find a way to compute ICRC after we used the UDP tunnel API which build the packets.

Fix PSN accounting to be done based on spec

Need to fix PSN accounting to be done based on spec:

Change remarks accordingly
Start with very large ranges based on switch buffering and latency, e.g., 50K packets…
Remove rxe_max_skb_per_qp module param and keep define

Loopback does not work

Running rping -dvsP & rping -cdv -a 192.168.1.103 gives the following:
verbose
server
verbose
created cm_id 0x6079d0
created cm_id 0x607ff0
rdma_bind_addr successful
rdma_listen
cma_event type RDMA_CM_EVENT_ADDR_RESOLVED cma_id 0x607ff0 (parent)
cma_event type RDMA_CM_EVENT_ROUTE_RESOLVED cma_id 0x607ff0 (parent)
rdma_resolve_addr - rdma_resolve_route successful
created pd 0x60a810
created channel 0x60a830
created cq 0x60a850
created qp 0x60a900
rping_setup_buffers called on cb 0x606010
allocated & registered buffers...
cq_thread started.

$ rxe_cfg
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
eth0 yes virtio_net 1500 192.168.122.113
eth1 yes virtio_net 1500 192.168.1.103 rxe0 1024 (3)

Change xmit logic

Need to change xmit logic by doing the following:

Remove select queue stuff when trying to detect if queue is stopped.
Change send_finish to be like FCoE xmit function.
Remove queue stopped definition and handle every TX error that could occur in rxe_arbiter

remove rxe_max_pkt_per_ack module parameter

Need to remove rxe_max_pkt_per_ack module parameter and instead define default value.

softroce / rxe-dev Goto Github PK

rxe-dev's Introduction

rxe-dev's People

Stargazers

Watchers

Forkers

rxe-dev's Issues

include <rdma/ib_rxe.h>

Recommend Projects

Recommend Topics

Recommend Org