libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 -------------------------------------------------------------------------- Failed to create a queue pair (QP): Hostname: groot Requested max number of outstanding WRs in the SQ: 1 Requested max number of outstanding WRs in the RQ: 2 Requested max number of SGEs in a WR in the SQ: 1023 Requested max number of SGEs in a WR in the RQ: 1023 Requested max number of data that can be posted inline to the SQ: 0 Error: Success Check requested attributes. -------------------------------------------------------------------------- -------------------------------------------------------------------------- Open MPI has detected that there are UD-capable Verbs devices on your system, but none of them were able to be setup properly. This may indicate a problem on this system. You job will continue, but Open MPI will ignore the "ud" oob component in this run. Hostname: groot -------------------------------------------------------------------------- libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 -------------------------------------------------------------------------- WARNING: No preset parameters were found for the device that Open MPI detected: Local host: groot Device name: i40iw0 Device vendor ID: 0x8086 Device vendor part ID: 14289 Default device parameters will be used, which may result in lower performance. You can edit any of the files specified by the btl_openib_device_param_files MCA parameter to set values for your device. NOTE: You can turn off this warning by setting the MCA parameter btl_openib_warn_no_device_params_found to 0. -------------------------------------------------------------------------- -------------------------------------------------------------------------- No OpenFabrics connection schemes reported that they were able to be used on a specific port. As such, the openib BTL (OpenFabrics support) will be disabled for this port. Local host: groot Local device: i40iw0 Local port: 1 CPCs attempted: rdmacm, udcm -------------------------------------------------------------------------- libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 libi40iw-i40iw_ucreate_qp: failed to create QP, unsupported QP type: 0x4 [groot.astro.sunysb.edu:30463] 16 more processes have sent help message help-oob-ud.txt / create-qp-failed [groot.astro.sunysb.edu:30463] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages [groot.astro.sunysb.edu:30463] 16 more processes have sent help message help-oob-ud.txt / no-ports-usable [groot.astro.sunysb.edu:30463] 31 more processes have sent help message help-mpi-btl-openib.txt / no device params found [groot.astro.sunysb.edu:30463] 15 more processes have sent help message help-mpi-btl-openib-cpc-base.txt / no cpcs for port amrex::Abort::14::GPU last error detected in file tmp_build_dir/s/3d.pgi.MPI.CUDA.EXE/AMReX_Random.cpp line 313: out of memory !!! SIGABRT amrex::Abort::15::GPU last error detected in file tmp_build_dir/s/3d.pgi.MPI.CUDA.EXE/AMReX_Random.cpp line 313: out of memory !!! SIGABRT amrex::Abort::10::GPU last error detected in file tmp_build_dir/s/3d.pgi.MPI.CUDA.EXE/AMReX_Random.cpp line 313: out of memory !!! SIGABRT amrex::Abort::2::GPU last error detected in file tmp_build_dir/s/3d.pgi.MPI.CUDA.EXE/AMReX_Random.cpp line 313: out of memory !!! SIGABRT amrex::Abort::12::GPU last error detected in file tmp_build_dir/s/3d.pgi.MPI.CUDA.EXE/AMReX_Random.cpp line 313: out of memory !!! SIGABRT amrex::Abort::8::GPU last error detected in file tmp_build_dir/s/3d.pgi.MPI.CUDA.EXE/AMReX_Random.cpp line 313: out of memory !!! SIGABRT amrex::Abort::6::GPU last error detected in file tmp_build_dir/s/3d.pgi.MPI.CUDA.EXE/AMReX_Random.cpp line 313: out of memory !!! SIGABRT amrex::Abort::0::GPU last error detected in file tmp_build_dir/s/3d.pgi.MPI.CUDA.EXE/AMReX_Random.cpp line 313: out of memory !!! SIGABRT See Backtrace.15 file for details See Backtrace.0 file for details See Backtrace.2 file for details See Backtrace.6 file for details See Backtrace.8 file for details See Backtrace.14 file for details See Backtrace.12 file for details See Backtrace.10 file for details -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 15 in communicator MPI_COMM_WORLD with errorcode 6. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- [groot.astro.sunysb.edu:30463] 7 more processes have sent help message help-mpi-api.txt / mpi-abort