Giter Club home page Giter Club logo

pydiskcmd's Introduction

Important

pydiskcmd remove pydiskhealthd in version 0.3.0, it may become a separate project in the future. For the reason:

If you still want to use it, keep your tool version less than 0.3.0.

pydiskcmd

pydiskcmd is a disk command tool for python3. It can send command to SATA/SAS/NVMe disk, as also as monitor the disk health.

In Linux, there is some tools to handle disk, like hdparm,smartctl,nvme-cli and etc. But I still hope to develop a tool to cover all the sata,sas,nvme disks. It should be easily installed and should be able to send raw commands to target disks, provide a high-level API to build raw command with different protocal. Besides, it could monitor the health of the disks, especially take full advantage of NVMe(which offer a better monitoring mechanism).

While in Windows, rarely find out an open source user level disk tool. I hope it is convenient to handle a disk in windows as if it is in Linux.

This project is more of a code collection, it was born and grow in the open source projects from github.

Why pydiskcmd

The pydiskcmd is easily installed. In most case, all you need is the python environment( sometimes python-devel is need) and python module Cython. Benefit from python (a cross-platform and high-level programming language) and OS level ioctl support, installing pydiskcmd Does Not need to configure the compiling environment(like C/C++ code). The long-term work in the development is to reduce the sofware dependency, to make installation as simple as possible.

The pydiskcmdlib provides a flexible and friendly middle-level API, as well as some low-level components to use. With the help of it, user could build their own raw command, send it to device and get the result. To archive this goal, more detailed documentation may be considered in the future. Whether you hope to test your storage device in protocal test or want to develop a disk utility tool, pydiskcmdlib will be a good choice.

pydiskcmdcli(pynvme/pysata/pyscsi included) is different with hdparm,sdparm and etc. The above tools is more like a command utility for a certain purpose, the pydiskcmdcli provides raw-command from spec without union. For example, the user cannot directly set the write-caching by the client tool, the equivalent command will be set-feature or mode-select.

Disadvantages

Compared to the compiled language, python code may take up more time and memory to run, and need more size in packaged program(i.e. program packaged by pyinstaller). Most of the time is consumed by loading the code in first use.

Test item Used Time(milliseconds) Description
Load pydiskcmdlib 35.99839 Only load module pydiskcmdlib
Load pydiskcmdcli 36.59582 Only load module pydiskcmdcli
NVMe.id_ctrl() 0.138521 run NVMe.id_ctrl() after pydiskcmdlib loaded
pynvme id-ctrl 69 run pynvme id-ctl , include print data

The test uses CentOS Stream 9 with python 3.9, and get the average value in 5 times.

License

pydiskcmd is distributed under LGPLv2.1. Please see the LICENSE file for the full license text.

Getting the sources

The module(source) is hosted at https://github.com/jackeichen/pydiskcmd

You can use git to checkout the latest version of the source code using:

$ git clone [email protected]:jackeichen/pydiskcmd.git

It is also available as a downloadable zip archive from:

https://github.com/jackeichen/pydiskcmd/archive/master.zip

Support List

OS arc SCSI ATA NVME
CentOS/RHEL 7.6 x64 Y Y Y
CentOS/RHEL 8.4 x64 Y Y Y
RHEL 9.1 x64 Y Y Y
SUSE 15 SP5 x64 Y Y Y
Ubuntu 22.04 x64 Y Y Y
Debian 10 x64 Y Y Y
Windows 10 Pro x64 Y Y Y
Windows 11 x64 Y Y Y
Windows Server 2019 x64 Y Y Y
Windows Server 2022 x64 Y Y Y

Y: support, N: Non-support, D: Developing, T: Under Testing

Note:

* Only some of the commands are tested, Do Not guarantee all the other commands work;
* This tool should work in Linux&Windows, but may be incompatible in OS other than this Support List;
* Windows only support some of the get-feature/get-log/idenfity commands;
* Support Direct-Connection/HBA Mode/JBOD Mode, RAID Mode is not support.

Building and installing

Sofware Requirements:

* python3(Required)

Python3 Module Requirements:

* Cython(Required)
* cython-sgio(Optional, can skip, but need latest version from github if used in linux)

Online build and install from the repository:

$ git clone [email protected]:jackeichen/pydiskcmd.git
$ pip install .

After your installation, you can use command to enable or update Linux Bash Completion for command pynvme&pysata&pyscsi(Only for Linux):

$ pynvme cli-autocmd

You can uninstall it by run:

$ pip uninstall pydiskcmd

Usage

Three executable programs should be added to environment variables after installation.

pynvme

It is a program similar to nvme-cli, with some limitted commands inside. Use bellow command to get help:

$ pynvme help
pynvme-0.3.0
usage: pynvme <command> [<device>] [<args>]

The '<device>' may be either an NVMe character device (ex: /dev/nvme0) or an
nvme block device (ex: /dev/nvme0n1) in Linux, while PhysicalDrive<X> in Windows.

The following are all implemented sub-commands:
  list                  List all NVMe devices and namespaces on machine
  list-subsys           List nvme subsystems
  list-ns               Send NVMe Identify List, display structure
  list-ctrl             Send NVMe Identify Controller List, display structure
  id-ctrl               Send NVMe Identify Controller
  id-ns                 Send NVMe Identify Namespace, display structure
  id-uuid               Send NVMe Identify UUID List, display structure
  create-ns             Creates a namespace with the provided parameters
  delete-ns             Deletes a namespace from the controller
  attach-ns             Attaches a namespace to requested controller(s)
  detach-ns             Detaches a namespace from requested controller(s)
  get-log               Generic NVMe get log, returns log in raw format
  smart-log             Retrieve SMART Log, show it
  error-log             Retrieve Error Log, show it
  commands-se-log       Retrieve Commands Supported and Effects Log, and show it
  fw-log                Retrieve FW Log, show it
  sanitize-log          Retrieve Sanitize Log, show it
  self-test-log         Retrieve the SELF-TEST Log, show it
  telemetry-log         Retrieve the Telemetry Log, show it
  persistent-event-log  Get persistent event log from device
  reset                 Resets the controller
  subsystem-reset       Resets the subsystem
  fw-download           Download new firmware
  fw-commit             Verify and commit firmware to a specific slot
  get-feature           Get feature and show the resulting value
  set-feature           Set a feature and show the resulting value
  format                Format namespace with new block format
  sanitize              Submit a sanitize command
  device-self-test      Perform the necessary tests to observe the performance
  pcie                  Get device PCIe status, show it
  show-regs             Shows the controller registers or properties. Requires character device
  flush                 Submit a flush command, return results
  read                  Submit a read command, return results
  verify                Submit a verify command, return results
  write                 Submit a write command, return results
  get-lba-status        Submit a Get LBA Status command, return results
  version               Shows the program version
  help                  Display this help

See 'pynvme help <command>' or 'pynvme <command> --help' for more information on a sub-command

The following are all installed plugin extensions:
  ocp                   OCP cloud SSD extensions
  vroc                  Windows NVMe VROC support

The following are pynvme cli management interface:
  cli-info              Shows pynvme information
  cli-autocmd           Enable or Update the command completion

See 'pynvme <plugin> help' for more information on a plugin

pysata

It is a sata command tool, to send ATA command to SATA Disk, with some limitted commands inside. Use bellow command to get help:

$ pysata help
pysata-0.3.0
usage: pysata <command> [<device>] [<args>]

The '<device>' is usually a character device (ex: /dev/sdb or physicaldrive1).

The following are all implemented sub-commands:
  list                        List all SATA devices on machine
  check-PowerMode             Check Disk Power Mode
  accessible-MaxAddress       Send Accessible Max Address command
  identify                    Get identify information
  self-test                   Start a disk self test
  set-feature                 Send set feature to device
  trusted-receive             Send trusted receive to device
  smart                       Get smart information
  smart-return-status         Get the reliability status of the device
  read-log                    Get the GPL Log and show it
  smart-read-log              Get the smart Log and show it
  standby                     Send standby command
  read                        Send a read command to disk
  write                       Send a write command to disk
  flush                       Send a flush command to disk
  trim                        Send a trim command to disk
  download_fw                 Download firmware to target disk
  version                     Shows the program version
  help                        Display this help

The following are pysata cli management interface:
  cli-info                    Shows pysata information
  cli-autocmd                 Enable or Update the command completion

See 'pysata help <command>' or 'pysata <command> --help' for more information on a sub-command

pyscsi

It is a scsi command tool, to send scsi command to SAS Disk, with some limitted commands inside. Use bellow command to get help:

$ pyscsi help
pyscsi 0.3.0
usage: pyscsi <sub-command> [<device>] [<args>]

The '<device>' is usually a character device (ex: /dev/sdb or physicaldrive1).

The following are all implemented sub-commands:
  list                        List all SCSI devices on machine
  inq                         Send scsi inquiry command
  getlbastatus                Get LBA Status from target SCSI device
  readcap                     Read capacity from target SCSI device
  luns                        Send Report Luns commandc to target SCSI device
  mode-sense                  Send Mode Sense command to target SCSI device
  log-sense                   Send Log Sense command to target SCSI device
  cdb-passthru                Submit an arbitrary SCSI command, return results
  se-protocol-in              Submit SECURITY PROTOCOL IN command, return results
  sync                        Synchronize cache to non-volatile cache, as known as flush
  read                        Send a read command to disk
  write                       Send a write command to disk
  version                     Shows the program version
  help                        Display this help

The following are all installed plugin extensions:
  parse-cmd                   Parse the CDB and sense code

The following are pyscsi cli management interface:
  cli-info                   Shows pyscsi information
  cli-autocmd                Enable or Update the command completion

See 'pyscsi help <command>' or 'pyscsi <command> --help' for more information on a sub-command

Advanced Usage

You can find some examples about how to use this tool in the dir of pydiskcmd/examples/.

Build Your Own Command

Example to build and run your own NVMe command in Linux.

### nvme format command
## <NVMeCommand> is the wrapper of raw command data structure, you can find it in pydiskcmdlib/pynvme/nvme_command,
#  <build_command> is the methmod to build cdw data structure
##
from pydiskcmdlib.pynvme.nvme_command import NVMeCommand,build_int_by_bitmap
from pydiskcmdlib.pynvme.linux_nvme_command import IOCTLRequest
## for running your own command in device, and get the result.
from pydiskcmdlib.utils import init_device

CmdOPCode = 0x80 # nvme format command OP code, see nvme spec

class Format(NVMeCommand):
    _req_id = IOCTLRequest.NVME_IOCTL_ADMIN_CMD.value  # define yourself request ID, admin or nvme command

    def __init__(self, 
                 lbaf,            # the lbaf to format, see nvme spec
                 mset=0,          # the mset to format, see nvme spec
                 pi=0,            # the pi to format, see nvme spec
                 pil=1,           # the pil to format, see nvme spec
                 ses=0,           # the ses to format, see nvme spec
                 nsid=0xFFFFFFFF, # the nsid to format, see nvme spec
                 timeout=600000): # timeout(millisecond) in IOCTL request
        ## build command cdw10
        #  this is a key-value dict input:
        #    key: the name of value
        #    value: (the bit-map of value, Byte offset in DWord, value to set)
        ##
        cdw10 = build_int_by_bitmap({"lbaf": (0x0F, 0, lbaf), # the location of lbaf in cdw10, see nvme spec
                                     "mset": (0x10, 0 ,mset), # the location of mset in cdw10, see nvme spec
                                     "pi": (0xE0, 0, pi),     # the location of pi in cdw10, see nvme spec
                                     "pil": (0x01, 1, pil),   # the location of pil in cdw10, see nvme spec
                                     "ses": (0x0E, 1, ses)})  # the location of ses in cdw10, see nvme spec
        ##
        super(Format, self).__init__()
        # build command
        self.build_command(opcode=CmdOPCode,
                           nsid=nsid,
                           cdw10=cdw10,
                           timeout_ms=timeout)

cmd = Format(0, nsid=1) ## format namespace 1 to lbaf 0
with init_device('/dev/nvme1', open_t='nvme') as d: ## open a nvme device: /dev/nvme1
    d.execute(cmd)
## Get the command result-> 
print (cmd.cq_cmd_spec) # Command Specific Status Values, see nvme spec
SC,SCT = cmd.check_return_status() # Get Command Status Code and Status Code Type
print ("Command Status Code=%d, Status Code Type=%d" % (SC,SCT))

Example to build and run your own SATA command in Linux Or Windows.

### Send an Identify command to SATA Disk
## pydiskcmdlib send SATA command by scsi passthrough12 or passthrough16 command
## This will import a suitable SCSIDevice depends on your OS,
#  The SCSIDevice help to send the command to device and get the result from the device
from pydiskcmdlib.utils import init_device
## ATACommand16 is the wrapper of ATA command, it help to build your own command
#  You can read ACS-3 about the ATA command set, and
#  read SAT-5 to make out how to translate from ATA command to SCSI passthrough command
from pydiskcmdlib.pysata.ata_command import ATACommand16


class Identify16(ATACommand16): 
    def __init__(self):
        ##
        # count is not used by idedntify in ATA command set,
        # so use it in ATAPassthrouh16, for setting transfer length
        ##
        ATACommand16.__init__(self,
                              0,         # fetures field
                              1,         # count field
                              0,         # lba field
                              0,         # device field
                              0xec,      # command field
                              0x04,      # protocal field
                              2,         # t_length field
                              1)         # t_dir field 

cmd = Identify16()
with init_device("/dev/sdb", open_t='scsi') as d:
    d.execute(cmd, en_raw_sense=True)
## Get the Result
# Handle the Command execute sense data
#  SAT-5 to make out ata_return_descriptor
ata_return_descriptor = cmd.ata_status_return_descriptor
print ("Command return Status:", ata_return_descriptor.get("status"))
# Get the datain, that read from device 
print ("Identify data read from device is:")
print (cmd.datain)

Example to build and run your own SCSI command in Linux Or Windows. It is a little different from NVMe Or SATA, the methmod is from the project python-scsi.

from pyscsi.pyscsi.scsi_command import SCSICommand
from pyscsi.pyscsi.scsi_enum_command import mmc, sbc, smc, spc, ssc
from pydiskcmdlib.utils import init_device

class Read16(SCSICommand):
    ## You need define the _cdb_bits to build cdb

    _cdb_bits = {
        "opcode": [0xFF, 0],
        "rdprotect": [0xE0, 1],
        "dpo": [0x10, 1],
        "fua": [0x08, 1],
        "rarc": [0x04, 1],
        "lba": [0xFFFFFFFFFFFFFFFF, 2],
        "group": [0x1F, 14],
        "tl": [0xFFFFFFFF, 10],
    }

    def __init__(
        self, lba, tl, rdprotect=0, dpo=0, fua=0, rarc=0, group=0
    ):
        """
        initialize a new instance

        :param lba: Logical Block Address
        :param tl: transfer length
        :param rdprotect=0:
        :param dpo=0:
        :param fua=0:
        :param rarc=0:
        :param group=0:
        """
        ##
        # This command is sbc command->READ_16, usually get by inquiry command, and
        # and the device is 512 byte logical format.
        ##
        opcode = sbc.READ_16
        blocksize= 512
        ## Build command
        SCSICommand.__init__(self, opcode, 0, blocksize * tl)

        self.cdb = self.build_cdb(
            opcode=self.opcode.value,
            lba=lba,
            tl=tl,
            rdprotect=rdprotect,
            dpo=dpo,
            fua=fua,
            rarc=rarc,
            group=group,
        )
## send command: 4k read from LBA 0 to LBA 7
cmd = Read16(0, 8)
## Execute Command
with init_device("/dev/sdb", open_t='scsi') as d:
    d.execute(cmd)
## Get the result
print (cmd.datain)

Acknowledgements

Really appreciate the project python-scsi in github.

pcicrawler is a CLI tool to display/filter/export information about PCI or PCI Express devices and their topology.

smartie is a pure-python library for getting basic disk information such as model, serial number, disk health, temperature, etc...

Communicate with NVMe SSD using Windows' inbox device driver

NVMe management command line interface.

pyPCIe provides a quick way to read/write registers in PCIe Base Address Register (BAR) regions.

A python/construct wrapper for sgio to replace cython-sgio

Support

If any support or ideas, open an issue, or contact author by email: [email protected]

pydiskcmd's People

Contributors

gao7904 avatar jackeichen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

pydiskcmd's Issues

Why fails in pynvme plugin vroc?

Intel® VROC is an enterprise RAID solution that unleashes the performance of NVMe SSDs, enabled by a feature in Intel® Xeon® Scalable processors called Intel® Volume Management Device (Intel® VMD), an integrated controller inside the CPU PCIe root complex. Intel® VROC enables these benefits without the complexity, cost, and power consumption of traditional hardware RAID host bus adapter (HBA) cards placed between the drives and the CPU.

NVMe Disk behind VROC in Linux could be act as a standalone nvme disk, the user could find out the nvme disks and raid volumes. But in Windows the user cannot get a physical drive if it is a member in a raid volume.

I hope it could be done in Windows, but it is only avaliable under Intel NDA now. So the code could not be published here.

I will keep this issue open, for telling you this message.

Can pysata pyscsi be used under Windows?

Hello,
Is pysata pyscsi only available under linux?

I get an error when I use pysata pyscsi in Windows; pynvme works fine on Windows。

The following errors are reported:

from pyscsi.pyscsi.scsi_cdb_atapassthrough12 import ATAPassThrough12
ModuleNotFoundError: No module named 'pyscsi.pyscsi.scsi_cdb_atapassthrough12'

Any one who cares about the disk health monitor pydiskhealthd?

This project pydiskcmd will focus on providing a disk command libaray and user client tool in next version 3.0.0. Function of pydiskhealthd will be removed, and it may become an independent project at some time.

I want to know if anyone concerns about the change? Any ideas or viewpoint would be great.

Something wrong when this tool works in windows server 2019

Although some comands work well in windows 10 Pro, it is difficult to do development in different windows version.

In my test the tool Do Not work in windows server 2019, and need some time to debug on this. The main work is always in Linux for now!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.