README for sg3_utils
                        ====================
Introduction
------------
sg3_utils is a package of utilities originally written to send individual
SCSI commands to storage devices that used one of the SCSI command sets.
These utilities can be divided into three groups:
   - sg_raw: the user supplies the cdb (command descriptor block) and
     optionally the size of the data-in and data-out buffers
   - one command utilities: the majority of the utilities in this package
     send one SCSI command. Their names start with "sg_" while the
     remaining part of their name alludes to the command which is sent. For
     example, "sg_inq" sends the SCSI INQUIRY command. Some utilities in
     this group send one of a selection of commands, typically those
     commands have a lot it common (e.g. sg_write_x).
   - copy type utilities: sg_dd, sgp_dd and sgm_dd use the Unix dd command
     as a template. sg_xcopy sends the SCSI EXTENDED COPY command which in
     some cases can do offloaded copies. As well as copying some of these
     utilities can compare if two data segments held on disks are the same.

Platforms
---------
These utilities were written on Linux and should work from Linux kernel
(lk) 2.4 through to the current series 5. The third group ("copy type")
are only implemented on Linux, but a separate portable package/utility
called ddpt implements similar functionality. The first two groups are
implemented (i.e. ported) to Android, FreeBSD, Solaris and Windows. The
Windows port uses either a Cygwin or MinGW (plus Msys) build environment
(rather than Visual Studio).

Library
-------
Many of these utilities share a lot of code (e.g. SCSI error messages)
so a lot of repetition (potentially error prone) is saved by having a
library called libsgutils or some variation on that name. Distributions
(especially of Linux) have differing policies on how a library (and a
package) should be named. For that reason this package is sometimes
known as "sg3-utils" (i.e. the underscore is turned into a hyphen).
Various other packages use libsgutils. The library interface is not
altered from one package release, to the next, but the library interface
may be expanded. If a utility from one release is used with a libsgutils
from an earlier release, then the runtime linking may fail. Typically
package managers take care of these details so that runtime linking
errors should be rare.

Command Sets
------------
SCSI command sets are not the only storage command sets in wide use, there
are also ATA and NVMe command sets. There is a SCSI command set to
translate SCSI commands to ATA commands (called SAT: SCSI to ATA
Translation). SAT includes an ATA PASS-THROUGH SCSI command and sg_sat_*
utilities (there are four) are examples of using SAT. The SAS transport
(Serial Attached SCSI) can convey ATA commands through a SCSI/SAS domain
via its Serial ATA Tunnelled Protocol (STP).

NVMe command sets (e.g. Admin, NVM and MI) are relatively new. There was an
early paper on a SCSI to NVMe Translation Layer (SNTL) but it hasn't been
standardized. The sg_inq utility will send (and decode the response of) a
SCSI INQUIRY command if the underlying device is a SCSI device. If the
underlying device is a NVMe controller or namespace, then sg_inq will send
a NVMe Admin Identify command and decode the response. The sg_ses utility
(for SCSI Enclosure Services) also checks whether its underlying device is
SCSI or NVME. In the NVMe case, sg_ses translates the SCSI SEND DIAGNOSTIC
and READ DIAGNOSTIC RESULTS commands to the NVMe Management Interface (MI)
SES Send and SES Receive commands respectively. The output of the sg_ses
utility should be similar, irrespective of whether the "SES" device is
SCSI or NVMe.

The sg_raw utility may send NVMe Admin or NVM commands (as well as SCSI
commands). One difficulty with a command-line utility invoking NVME
commands is that those commands contain memory addresses for data-in (from
the storage device) or data-out (toward the storage device) transfers. See
the sg_raw manpage for how this difficulty is addressed.

Documentation
-------------
Manual pages ("manpages") are the primary method of utility documentation.
All utilities and scripts that are installed by this package have a
manpage. There are utilities in the examples, testing and utils
directories that are not installed and do not have manpages. Nearly
all utilities have runtime help, usually invoked with either the '-h'
short option or the '--help' long option. There is also an overarching
manpage called "sg3_utils". All manpages are placed in chapter 8 which
is for system administration commands/utilities.

The sg3_utils package and some more complex utilities have html pages:
   sg3_utils: https://sg.danny.cz/sg/sg3_utils.html
   sg_ses:    https://sg.danny.cz/sg/sg_ses.html
   sg_dd:     https://sg.danny.cz/sg/sg_dd.html

A tarball (and zip) of all the manpages from the previous release are
here:
   https://sg.danny.cz/sg/p/sg3_utils_man_html.tgz
   https://sg.danny.cz/sg/p/sg3_utils_man_html.zip

There is a html rendering of the sg3_utils manpage in the same directory
as this README file called sg3_utils.man8.html .

The previous README file is now called README.details plus there are
these OS specific files: README.freebsd , README.solaris , README.tru64
and README.win32 . To know the current state of the package the ChangeLog
file is the good reference.

The author's primary source code repository uses subversion and is on
the author's equipment (a RPi). One advantage of subversion is its
revision numbers which are simply integers starting at 1 and ascending.
For this package the current revision is 928 . The subversion repository
is mirrored in git (using "git svn" tools) here:
    https://github.com/doug-gilbert/sg3_utils


Douglas Gilbert
31st December 2021

                      README.details for sg3_utils
                      ============================
Introduction
============
This package contains low level command line utilities for devices that use
the SCSI command set. Originally the SCSI command set was associated
exclusively with the SCSI Parallel Interface (SPI) transport. SPI has now
almost been completely replaced by the Serial Attached SCSI (SAS) transport
which also accepts the SCSI command set. Additionally many other storage
related transports use the SCSI command set (amongst others); examples are
ATAPI devices (CD/DVDs and tapes), USB mass storage devices (including those
using the newer UAS[P]), Fibre Channel disks, IEEE 1394 storage devices (SBP
protocol), iSCSI, FCoE and SOP devices. Even NVMe which has its own command
set accepts SCSI commands in some contexts; one example is for enclosure
management where NVME-MI has SES Send and SES Receive commands. SES refers
to the SCSI Enclosure Services command set.

This package originally targeted the Linux SCSI subsystem. Since most
operating systems contain a SCSI command pass-through mechanism, many
utilities within this package have been ported. This README mainly
concentrates on Linux: see the README.freebsd file for the FreeBSD port,
README.solaris for the Solaris port, the README.tru64 file for the Tru64
(OSF) port and README.win32 for the Windows ports (of which there are two
variants).

Most utilities within the sg3_utils package work at the SCSI command level.
For example the sg_inq utility issues a SCSI INQUIRY command and decodes the
response. The COVERAGE file has a table containing a row for each SCSI
command issued by this package; to the right of each row is the utility
(sometimes more than one) that issue that SCSI command. The COVERAGE file
has a second table for ATA commands usage.

Some utilities interface at a slightly higher level, for example: sg_dd,
sgm_dd and sgp_dd. These are closely related to the Unix dd command and
typically issue a sequence of SCSI READ and WRITE commands to copy data.
These utilities are relatively tightly bound to Linux and are not ported to
other Operating Systems. A new utility called ddpt (in a package of the same
name) is more generic while still allowing a copy to be done in terms of
SCSI READ and WRITE commands. ddpt has been ported to other OSes.

License
=======
All utilities and libraries have either a "2 clause" BSD license or are
"GPL-2ed". The "2 clause" BSD license is taken from the FreeBSD project but
drops the last paragraph that directly refers to the "FreeBSD project".
That BSD license was updated from the "3 clause" to the newer "2 clause"
version on 20180119. To save space various source code files refer to a
file called "BSD_LICENSE" in the main, src and lib directories. The author's
intention is that users may incorporate all or part of the code in their work
as they please. Attribution is encouraged. Please check the code as other
contributors (apart from the author) may also have copyright notices. For a
list of contributors see the CREDITS file.


Description
===========
A web site supporting the sg3_utils package can be found at
https://sg.danny.cz/sg/sg3_utils.html . That page has a table of released
versions for download. The most recent release or beta of sg3_utils may
be found on this page: https://sg.danny.cz/sg in the News section.

The predecessor to this package was called sg_utils. It is described in
https://sg.danny.cz/sg/uu_index.html and old versions can be downloaded
from the Downloads section of https://sg.danny.cz/sg .

In the Linux 2.4 kernel series these utilities need to use the SCSI generic
(sg) driver to access SCSI devices. The name of this package (i.e. sg3_utils)
refers to version 3 of the SCSI generic (sg) driver which was introduced at
the beginning of the 2.4 Linux kernel series. Significantly this added a new
SCSI command interface structure (i.e. struct sg_io_hdr) that is more
flexible than the older "sg_header" structure found in the sg driver in the
2.2 and earlier Linux kernel series. The sg_io_hdr structure is also more
flexible than the awkward (and limiting) interface to the
SCSI_IOCTL_SEND_COMMAND ioctl supported by the Linux SCSI mid level. The
version 3 sg driver also added the SG_IO ioctl that is synchronous (i.e. it
issues the requested SCSI command and waits for the response (or a timeout)
before the ioctl returns to the user space program that invoked it). The
SG_IO ioctl is now supported in other parts of the Linux kernel in the 2.6
series.

In sg3_utils version 1.27 support has been added for the Linux bsg driver
which use the sg version 4 interface. There seems no point in renaming
this package sg4_utils. The existing utilities just silently support either.
Currently the source build must be able to see the /usr/include/linux/bsg.h
file. Then at run time the /proc/devices pseudo file needs to have an entry
for the bsg driver (appeared around lk 2.6.28). With this in place each
utility at run time checks the device it has been given and if it is a char
device whose major number matches the bsg entry in /proc/devices then the
sg v4 interface is used. Otherwise the sg v3 interface is used.

Utilities that wish to use the asynchronous SCSI command interface (i.e. via
a write() read() sequence) or issue special "commands" (e.g. bus and device
resets) still need to use the Linux sg driver. Note that various
drivers (e.g. cdrom/sr) have different open() flag and permissions policies
that the user may need to take into account.

If users have problems or questions about them please contact the author.
Documentation for the Linux sg device driver can be found at:
https://sg.danny.cz/sg/p/sg_v3_ho.html . This is written in DocBook and the
original xml can be found in the same directory with the ".xml" extension.
Postscript and pdf renderings are also in that directory. Older documentation
for the sg version 3 driver can be found at:
https://sg.danny.cz/sg/p/scsi_generic_v3.txt .

To save the repetition of common code (e.g. SCSI error processing) and
reduce the size of the executable files, a shared library called
libsgutils<num>.so (its Linux name) is created during the build process.
That library is built from the contents of the include and lib
subdirectories. The header files in the include subdirectory can be seen
as the API of libsgutils and are commented with that in mind. The SCSI
pass-through code for the supported operating systems is found in the lib
subdirectory with names like sg_pt_linux.c and sg_pt_win32.c .

Various distributions (of Linux mainly) distribute sg3_utils as 3
installable packages. One is a package containing the shared library
discussed above (e.g. libsgutils2-2_1.33-0.1_i386.deb). A second package
contains the utilities (e.g. sg3-utils_1.33-0.1_i386.deb) and depends on the
first package). Finally there is an optional package that contains header
files and a static library (e.g. libsgutils2-dev_1.33-0.1_i386.deb). This
final package is only needed to build other packages (e.g. sdparm) that
wish to use the sg3_utils shared library.

All the utilities in the src subdirectory have "man" pages that are
placed in the doc subdirectory. There is also a sg3_utils (8) man page that
summarizes common facilities including exit statuses. Additional
information (including each utility's version number) can be found towards
the top of each ".c" file corresponding to the utility name.

The sg driver in Linux can be seen as having 3 distinct versions:

   v1   lk < 2.2.6     sg_header based relatively unchanged since 1992
   v2   lk >= 2.2.6    enhanced sg_header interface structure [1999/4/16]
   v3   lk >= 2.4      additional sg_io_hdr interface structure [2001/1/4]
   v3   lk >= 2.6      same interface as found in lk 2.4 [2.6.0: 2003/12/18]

and the bsg driver supports the sg v4 interface and was added around
lk 2.6.28 . This package is targeted at "v3" and "v4". Another package called
"sg_utils" is targeted at "v2" and to a lesser extent "v1". The "sg_utils"
package has a subset of the utilities found in this package.

In Linux some sg driver ioctls (notably SG_IO) are defined for many block
devices in lk 2.6 series. In practice this means all SCSI block devices,
ATAPI block devices (mainly CD, DVD and BD optical devices) but _not_ ATA
disks, depending on which kernel configuration options, can be accessed by
the utilities in this package. SATA disks that use the libata kernel library
(or some other SCSI to ATA Translation (SAT) Layer (SATL)) accept SCSI
commands and thus are supported. Support for the SG_IO as been added to the
scsi tape driver (st) in lk 2.6.6 .

In the src directory the bulk of the utilities are written in relatively
clean POSIX compliant C code with Linux specific system calls and structures
removed and placed in Linux specific files in the lib directory. A small
number of utilities in the src directory do contain Linux specific logic
and are not ported to other OSes (e.g. sg_dd). One utility, sg_scan, has
two separate implementations, one for Linux (sg_scan_linux.c) and one for
Windows (sg_scan_win32.c). The src-lib directory split approach allows
FreeBSD, Solaris, Tru64 and Windows specific code to be isolated to a few
files in the lib directory whose interfaces match those of the Linux
specific code.

Darwin is not supported because the Apple folks do not want to give their
users a pass-through SCSI interface. The author has read about creative
hackers using a VM containing a real OS to circumvent the Apple restriction.

C standard is C11
==================
The C code in this package is written for portability rather than speed.
It assumes a level of C99 compliance (the C standard prior to C11) and
favours POSIX system and library calls over OS specific calls.

The C code is written in a C++ friendly way and is checked from time to
time that it compiles clean with C++. To accommodate C++ certain C99
constructs such as designated initializers cannot be used. To build
with C++, C++11 (i.e. the C++ standard from 2011) or later is required.
Finding a common C and C++ syntax for zeroing stack variables (including
aggregates) may need to wait until C23 allows this syntax:
   struct example_t ex1 {};
which C++ introduced in C++11.  In the meantime the SG_C_CPP_ZERO_INIT
define (hack) does this.

The author has not seriously attempted to build this code on MSVC (aka
Visual Studio). There are a few roadblocks (that may be overcome in the
future) that include MSVC being basically a C++ compiler, not a C/C++
compiler. For some reason MSVC only claims C89 compliance (i.e. the first
C standard from 1989). MSVC 2013 and 2015 are moving closer to C99
compliance and may be sufficient to compile this package. Another problem
is the assumption of the availability of basic Unix system calls such as
open(). Nearly 20 years ago Microsoft indicated (promised ?) that it
would move in the direction of POSIX compliance, but very little ever
happened. "Talk is cheap, there should be a tax on it."

Building
========
This package is designed to be built with the usual:
    "./configure ; make ; make install"
sequence. In some situations that may need to be prefixed by a call to
the "./autogen.sh" script which invokes autoconf and automake. That in turn
may require packages containing those utilities to be installed. The
libtool utility is also required. Naturally a C compiler is required
and due to the vagaries of libtool a C++ compiler also.

The "./configure" takes many command line options with the defaults
being usually sufficient to start with. One quirk is that the location
of the installation is under the /usr/local directory. So the sg_inq
utility will be installed at /usr/local/bin/sg_inq . This is controlled
by the "--prefix=<directory>" option which defaults to
"--prefix=/usr/local". As an example to install the executables in /usr/bin
and disable the creation of the shared library (libsgutils<num>.so) this
invocation could be used: "./configure --prefix=/usr --disable-shared".
To reduce the size of an executable as well try this:
"./configure --prefix=/usr --disable-shared --disable-scsistrings".
Also --disable-shared will produce (relatively) "static" executables in
the src directory that are easier to debug. And
"./configure --enable-debug" will compile with more debug type options,
including more compiler checks and defining "DEBUG" within the src and
lib source files. Most utilities in the src directory set '-vv' (i.e.
equivalent to calling "--verbose" twice) when "DEBUG" is set.

In Linux there are package build files for "rpm" based and for "deb" based
systems. The 'sg3_utils.spec' file in the main directory can be used like
this: 'rpmbuild -ba sg3_utils.spec' in a rpmbuild tree SPECS directory.
To cross build or make a more widely distributable package then the --target
option may be useful: 'rpmbuild --target=i386 -ba sg3_utils.spec' or
'rpmbuild --target=x86_64 -ba sg3_utils.spec' . The sg3_utils.spec file
in the main directory targets Red Hat systems, an alternative "spec" file
for Suse systems has been placed under the 'suse' directory.

The 'build_debian.sh' script should build several "deb" packages and place
them in the parent directory. In debian based systems doing
a 'apt-get install build-essential' is one way to get most of build
environment needed if it has not already been loaded. There are now some
problems with this script and the superseded Debian 4.0 ("etch"). See
debian/README.debian4 for a workaround. Amongst other things debian
builds are sensitive to the value in the debian/compat file. If it
contains "7" then it works on lenny and gives warning on squeeze (but
fails on the earlier etch).

Warning
=======
Many devices use SCSI command sets over transport protocols not normally
associated with SCSI (as defined at https://www.t10.org ). Some of these
devices react poorly (e.g. lock up) when sent SCSI commands that they don't
support. Even sending a supported SCSI command with a field set to an
unexpected value can cause problems. [The author is talking about billions
of USB devices with horrible SCSI implementations.]

For example, all "SCSI" devices must support the INQUIRY command which the
SCSI-2 standard says should request a 36 byte response. However later SCSI
standards (e.g. SPC-2) have increased that length but some SCSI devices lock
up when they receive a request for anything other than a 36 byte response.

Any well implemented "SCSI" device should react sensibly when a utility in
sg3_utils sends a SCSI command that it doesn't support. Unfortunately this
cannot be guaranteed.

Prior to lk 2.6.29 USB mass storage limited sense data to 18 bytes which
caused problems for certain types of descriptor based sense data. An
example of this is the SCSI ATA PASS-THROUGH command with the CK_COND bit
set.


Utilities
=========
Here is list in alphabetical order of utilities found in the 'src'
subdirectory of the sg3_utils package:
    sginfo, sg_bt_ctl, sg_compare_and_write, sg_copy_results, sgm_dd, sgp_dd,
    sg_dd, sg_decode_sense, sg_emc_trespass, sg_format, sg_get_config,
    sg_get_elem_status, sg_get_lba_status, sg_ident, sg_inq, sg_logs,
    sg_luns, sg_map, sg_map26, sg_modes, sg_opcodes, sg_persist, sg_prevent,
    sg_raw, sg_rbuf, sg_rdac, sg_read, sg_read_attr, sg_readcap,
    sg_read_block_limits, sg_read_buffer, sg_read_long, sg_reassign,
    sg_referrals, sg_rem_rest_elem, sg_rep_density, sg_rep_pip, sg_rep_zones,
    sg_request, sg_reset, sg_rmsn, sg_rtpg, sg_safte, sg_sanitize,
    sg_sat_identify, sg_sat_phy_event, sg_sat_read_gplog, sg_sat_set_features,
    sg_scan, sg_seek, sg_senddiag, sg_ses, sg_ses_microcode, sg_start,
    sg_stpg, sg_stream_ctl, sg_sync, sg_test_rwbuff, sg_timestamp, sg_turs,
    sg_unmap, sg_verify, sg_vpd, sg_write_buffer, sg_write_long,
    sg_write_same, sg_write_verify, sg_write_x, sg_wr_mode, sg_xcopy, sg_zone,
    sg_z_act_query

Each of the above utilities depends on header files found in the 'include'
subdirectory and library code found in the 'lib' subdirectory. Associated
man pages are found in the 'doc' subdirectory. Additional programs found
in the 'archive', 'examples' and 'utils' subdirectories in not build by the
top level build infrastructure. Linux binary distributions of the sg3_utils
package (e.g. "rpm" and debian packages) typically contain the shared
library, the utilities found in the 'src' subdirectory, their associated man
pages and some documentation files (e.g. README, INSTALL, CREDITS, COPYING
and COVERAGE). See the INSTALL file for generic instructions about building
with autotools (e.g. ./configure ).

Man pages can be read (without building and installing the package) by
going to the 'doc' subdirectory and executing something like this:
 $ man ./sg_dd.8

To see which SCSI commands (and ATA commands) are used by these utilities
refer to the COVERAGE file.

Here is a list in alphabetical order of utilities found in the 'examples'
subdirectory:
  - sg_excl, scsi_inquiry, sg_sat_chk_power, sg__sat_identify,
    sg__sat_phy_event, sg__sat_set_features, sg_sat_smart_rd_data,
    sg_simple1, sg_simple2, sg_simple3, sg_simple4, sg_simple5,
    sg_simple16

Also in that subdirectory is a script to test sg_persist, an example data
file for sg_persist (called "transport_ids.txt") and an example data file for
sg_reassign (called "reassign_addr.txt"). There are several scripts
for 'sg_senddiag -pf -raw=-' that will put some SAS disk phys into
a "compliant jitter tolerance pattern" (CJTPAT).

The 'testing' subdirectory contains source and a Makefiles to test
kernel pass-through and associated drivers, mainly for Linux. There is
both C code (with the extension ".c") and C++ code (with the extension
".cpp"). There is a "Makefile" to build the C + C++ code. The Makefile
depends on some object files from the "lib" subdirectory. So a sequence
like this may be required prior to invoking make: "cd <top_of_package> ;
./configure ; cd lib ; make ; cd ../testing".

Here is a list in alphabetical order of utilities found in the 'testing'
subdirectory:
  - bsg_queue_tst, sgh_dd (C++), sg_iovec_tst, sg_queue_tst, sg_sense_tst,
    sg_tst_async (C++), sg_tst_context (C++), sg_tst_excl (C++),
    sg_tst_excl2 (C++), sg_tst_excl3 (C++)

The 'utils' subdirectory contains source and a Makefile to build "hxascdmp"
which accepts binary data from stdin (or a file on the command line) and
outputs an ASCII-HEX and ASCII representation of it. It is similar to the
Unix od command. There is also code to sg_chk_asc.c which checks a given
text file (typically a copy of https://www.t10.org/lists/asc-num.txt ) and
checks it against the asc/ascq text strings held in sg_lib_data.c .

The 'doc' subdirectory contains a README file containing the urls of
various related documents.

The 'scripts' subdirectory contains some Bourne (bash) shell scripts that
rely on utilities in the main directory. One script uses the sdparm utility.
These scripts are described in the scripts/README file and have usage
messages.


Notes for utilities without man pages
=====================================
These utils are found in the 'examples' subdirectory.

The "scsi_inquiry" program shows the use of the SCSI_IOCTL_SEND_COMMAND
ioctl to send a SCSI INQUIRY command. That ioctl() is supported by the
SCSI sub system mid level and so is common to all sd, sr, st and sg devices.
That ioctl is deprecated in the lk 2.6 series. This program has been placed
in the "examples" subdirectory.

"sg_simple1" and "sg_simple2" are example programs demonstrating calls
to the SCSI INQUIRY and TEST UNIT READY commands. They only differ in their
error processing: sg_simple1 uses sg_lib.[hc] for error processing while
sg_simple2 does its own more primitive checks.

"sg_simple3" tests out user space scatter gather added to the version 3
sg driver.

"sg_simple4" shows the INQUIRY command using mmap-ed IO to obtain its
response buffer.

"sg_simple5" also sends and INQUIRY and TEST UNIT READY commands. It
uses the generic pass through mechanism based on sg_pt.h . It will
currently build in Linux and FreeBSD (with "make -f Makefile.freebsd").
It has extensive error checking code.

"sg_simple16" attempts to send a 16 byte SCSI command, READ_16, to the
scsi device. This is only supported for lk >= 2.4.15 and for adapter
drivers that indicate that they have 16 byte CDB capability (otherwise
DID_ABORT will appear in the host_status).

"sg_sat_chk_power" attempts to push an ATA CHECK POWER MODE command
through the SAT-defined ATA PASS_THROUGH (16) SCSI command. That
ATA command needs to read the "FIS" registers after the command is
completed which involves using the ATA Status Return (sense data)
descriptor (as defined in SAT).

"sg_sat_smart_rd_data" attempts to push an ATA SMART/READ DATA command
through the SAT-defined ATA PASS_THROUGH (16) SCSI command. If
successful, the 256 word (512 byte) response is output.

"sg_tst_excl" and "sg_tst_excl2" use multiple threads to bombard the
given device with O_EXCL open flags, so only one should succeed at a
time. While holding O_EXCL control a thread attempts a double increment
on an integer in the given LBA. If the integer starts even (after the
first read) then it should remain even if the O_EXCL flag is doing its job.
The "sg_tst_excl" variant uses the Linux SG_IO v3 interface while the
"sg_tst_excl2" uses the more generic sg_pt infrastructure.

"sg_tst_excl3" is a variant of "sg_tst_excl2". "sg_tst_excl3" only does
the double increment from the first thread, each time using O_EXCL on
open. The remaining threads check the value is even, each time doing
an open without the O_EXCL flag.

"bsg_queue_tst" sends an INQUIRY command via the Linux SG_IO v4 interface
which is used by the bsg driver. So it will take device names like
"/dev/bsg/6:0:0:0". It tests if sending repeated INQUIRYs with
the BSG_FLAG_Q_AT_HEAD or BSG_FLAG_Q_AT_TAIL flag makes any difference.

"sg_tst_async" is a test harness for the Linux sg driver. It is multi
threaded, submitting either TEST UNIT READY, READ(16) or WRITE(16) SCSI
commands asynchronously. Each thread opens a file descriptor and submits
those commands up to the queue limit (sg driver has a per file descriptor
queue limit of 16). Multiple threads doing the same thing act as a
multiplier to that queue limit.


NVME Support
============
Firstly the author has no intention of extending this package to contain
general purpose NVMe utilities. That leaves the areas where SCSI overlaps
with NVMe. There was a SCSI to NVMe Translation Layer (SNTL) driver in the
Linux kernel based on a white paper from NVM Express. Intel has withdrawn
that driver and T10 (SCSI) and NVM Express have made no further attempts
to standardize a SNTL. Given the SCSI to ATA Translation Layer (SATL) which
is standardized by T10, it is pretty clear what a SNTL should do.

The NVMe Management Interface (NVME-MI) committee have decided to use SES-3
standard from T10 via the newly added SES Send and SES Receive MI commands.
So the sg_ses utility and this package's library have been extended to use
these commands when a NVMe device (typically a disk enclosure) is detected.
This has been tested by a disk vendor who is happy with the results. Other
user reports are welcome as the author does not have equipment to test
this.

Other utilities in this package that use the SES Send and Receive commands,
or the SNTL in the library are sg_senddiag, sg_inq, sg_raw and sg_readcap.


Command line processing
=======================
These utilities can be divided into 3 groups when their handling of command
line arguments is considered:
  - ad hoc, typically in a short form only, sometimes longer (e.g.
    "sg_logs -pcb /dev/sdc")
  - inspired by the dd Unix command (e.g. sg_dd, sgm_dd, sgp_dd, sg_read)
  - recent utilities use "getopt_long" (see "man getopt_long")
    type command lines. These have short form (starting with "-")
    and corresponding longer form (starting with "--") options.

The older utilities that use ad hoc options, in alphabetical order:
  - sg_emc_trespass, sginfo(1/2), sg_inq, sg_logs, sg_map, sg_modes,
    sg_opcodes, sg_rbuf, sg_rdac, sg_readcap, sg_reset, sg_scan (Linux),
    sg_senddiag, sg_start, sg_test_rwbuf, sg_turs
In sg3_utils version 1.23 the following utilities from this group were
converted to have a dual getopt_long/ad_hoc interface, defaulting to
the getop_long interface:
  - sg_inq, sg_logs, sg_modes, sg_opcodes, sg_rbuf, sg_readcap,
    sg_senddiag, sg_start, sg_turs
These can be switched back to the older (backward compatible) ad hoc
interface by defining the SG3_UTILS_OLD_OPTS environment variable
or using '-O' as the first command line option.

The more recent utilities that use "getopt_long" only are:
  - sg_bt_ctl, sg_compare_and_write, sg_decode_sense, sg_format,
    sg_get_config, sg_get_lba_status, sg_ident, sg_luns, sg_map26,
    sg_persist, sg_prevent, sg_raw, sg_read_attr, sg_read_block_limits,
    sg_read_buffer, sg_read_long, sg_reassign, sg_referrals, sg_rep_pip,
    sg_rep_zones, sg_requests, sg_rmsn, sg_rtpg, sg_safte, sg_sanitize,
    sg_sat_identify, sg_sat_phy_event, sg_sat_read_gplog,
    sg_sat_set_features, sg_scan(w), sg_seek, sg_ses, sg_ses_microcode,
    sg_stpg, sg_stream_ctl, sg_sync, sg_test_rwbuf, sg_timestamp, sg_unmap,
    sg_verify, sg_vpd, sg_write_buffer, sg_write_long, sg_write_same,
    sg_write_verify, sg_write_x, sg_wr_mode, sg_zone, sg_z_act_query


Dangerous code
==============
This C code snippet:
    unsigned char uc = 0x80;
    uint64_t ull;
    ull = (uc << 24);
Somewhat surprisingly sets ull to:
    ull: 0xffffffff80000000
This result is due to the 'unary conversion' of uc to a (32 bit signed)
'int' before the shift. The resultant type from the shift is also an int
and it has its top bit set so there is sign extension when it is assigned
into a 64 bit unsigned integer. Making sure there is no conversion to 'int'
solves the problem. In this case if uc is declared as unsigned int the
result will be as expected (i.e. 0x80000000).


Bypassing the somewhat dangerous shift operators
================================================
The shift operators in C are "<<" and ">>". They can be dangerous (as shown
in the above section) or tedious and hence error prone to use. However they
are often needed to cope with the translation of integers on the host OS to
the corresponding representation within a SCSI command or parameter data
moved to or from a SCSI device. The Logical Block Address (LBA) is a good
example; it is either 32 or 64 bits long typically (i.e. 4 or 8 bytes
respectively). The host machine representation may be big or little endian
and may prefer or require alignment to a particular memory address boundary
(e.g. module 4 (or in 'C' code: "(lba % 4) == 0")). For SCSI commands and
the parameter data moved to or from a SCSI device, the integer
representation is big endian and it is unaligned.

Recent versions of this package have replaced the explicit use of the C
shift operators with a group of functions modelled on those found in the
Linux kernel. These functions contain either "get_unaligned" or
"put_unaligned" in their names and are found in the asm/unaligned.h
header. This package contains the sg_unaligned.h header that implements
a similar set of functions. The current implementation favours correctness
over speed. The functions in the package use a "sg_" prefix but otherwise
use the same function name as the Linux kernel for the same action.

An example of the change made to a snippet of sg_write_buffer.c may
clarify this change. The old code was:

   wbufCmdBlk[3] = (unsigned char)((buffer_offset >> 16) & 0xff);
   wbufCmdBlk[4] = (unsigned char)((buffer_offset >> 8) & 0xff);
   wbufCmdBlk[5] = (unsigned char)(buffer_offset & 0xff);

and it has been replaced by:

   sg_put_unaligned_be24(buffer_offset, wbufCmdBlk + 3);

The Linux kernel only supplies "unaligned" functions for 16, 32 and 64
bit quantities. SCSI commands also have cases of 24 and 48 bit numbers
so sg_unaligned.h contains support for those plus a variant where the
byte length is passed as an argument.

The unaligned functions are inlined for speed (at the possible expense of
space) and now have specializations depending whether the host is big or
(more likely) little endian. These functions can be broken down to a
memcpy() and optionally a byte-swap for 16, 32 and 64 bit operations.
The memcpy() takes care of alignment while the byte-swap (bswap_16(),
bswap_32() and bswap_64() ) addresses integer endianness. If the host is
little endian and a little endian variant of the unaligned functions is
requested, then no byte-swap is required. These specializations can be
"compiled out" with this configure option: './configure
--disable-fast-lebe' in which case the classic "C shifting" technique is
used to implement all the unaligned functions.

Associated with the above change, fixed length integer types seem a better
fit for SCSI command and parameter integers than the traditional integer
types in the C language. Fixed length integer types were standardized in
C99 and require the inclusion of <stdint.h>. For example this means for
an integer that will represent a 64 bit LBA, to favour using "uint64_t"
over the "unsigned long long" type. Also "unsigned char" has mostly been
replaced by "uint8_t" as the 8 bit (unsigned) byte type; "char" is still
used for ASCII text.


Coding Style
============
Everyone has their own C/C++ coding style and the author is no different.
In terms of the GNU indent command:
     indent -i4 -il0 -nut -br -npcs -ncs -ce
is pretty close. That is similar to the Linux kernel coding style but
with 4 space indentations and no tabs.


Other SCSI and storage tools
============================
See https://sg.danny.cz/sg/tools.html


Douglas Gilbert		dgilbert@interlog.com
26th August 2022

Introduction
============
The FreeBSD port of sg3_utils contains those utilities that are _not_
specific to Linux. In some cases the FreeBSD camcontrol command supplies
similar functionality; for example 'sg_map' is similar to
'camcontrol devlist'.

The dd variants from the sg3_utils package (e.g. sg_dd) rely on too many
Linux idiosyncrasies to be easily ported. A new package called 'ddpt'
contains a utility with similar functionality to sg_dd and ddpt is available
for FreeBSD.

Supported Utilities
===================
Here is a list of utilities that have been ported:
    sg_bg_ctl
    sg_compare_and_write
    sg_decode_sense
    sg_format
    sg_get_config
    sg_get_elem_status
    sg_get_lba_status
    sg_ident
    sg_inq          [dropped ATA IDENTIFY DEVICE capability]
    sg_logs
    sg_luns
    sg_modes
    sg_opcodes
    sg_persist
    sg_prevent
    sg_raw
    sg_rdac
    sg_read_block_limits
    sg_read_buffer
    sg_read_long
    sg_readcap
    sg_reassign
    sg_referrals
    sg_rep_pip
    sg_rep_zones
    sg_requests
    sg_rmsn
    sg_rtpg
    sg_safte
    sg_sanitize
    sg_sat_identify
    sg_sat_phy_event
    sg_sat_set_features
    sg_seek
    sg_senddiag
    sg_ses
    sg_start
    sg_stpg
    sg_stream_ctl
    sg_sync
    sg_turs
    sg_verify
    sg_unmap
    sg_vpd
    sg_wr_mode
    sg_write_buffer
    sg_write_long
    sg_write_same
    sg_write_verify
    sg_write_x
    sg_zone

Most utility names are indicative of the main SCSI command
that they execute.  Some utilities are slightly higher level, for
example sg_ses fetches SCSI Enclosure Services (SES) status pages and
can send control pages. Each utility has a man page (placed in
section 8). An overview of sg3_utils can be found at:
https://sg.danny.cz/sg/sg3_utils.html .
A copy of the "sg3_utils.html" file is in the "doc" subdirectory.


The executables and library can be built from the source code in
the tarball and installed with the familiar
"./configure ; make ; make install" sequence. If this fails try
running the "./autogen.sh" script prior to that sequence. There
are generic instruction on configure and friend in the INSTALL file.

Some man pages have examples which use Linux device names which
hopefully will not confuse the FreeBSD users.

Device naming
=============
In FreeBSD disks have block names like '/dev/da0' with a corresponding
pass-through device name like '/dev/pass0'. Use this command:
"camcontrol devlist" to see that SCSI devices available. To list NVMe
devices: "nvmecontrol devlist" can be used. Any many, but not all
contexts, the device name can be used without the '/dev/' prefix.
FreeBSD is relatively unique in this respect and support for this
abbreviated form has been broken in this package and fixed in
sg3_utils release 1.46 .

Device naming for NVMe is a bit more complex. Controllers have names
like /dev/nvme0 and namespaces /dev/nvme0ns1 . Partitions are not
supported on /dev/nvme0ns1 type nodes. Instead there are /dev/nvd0
and /dev/nvd0p<m> where <m> is th partition number starting at 1.
The nvd driver (written by Intel) is not CAM compatible and has its
own utility nvmecontrol which has similar capabilities as camcontrol
has for CAM devices. In FreeBSD release 12 the nda driver was
introduced with names like /dev/nda0 and /dev/nda0n<m>. The difference
is that nda is CAM compatible. From the point of view of this package,
the nda driver is preferred as CAM supports NVMe command timeouts and
the error processing is more mature.

FreeBSD installation
====================
The traditional './configure ; make ; make install' sequence from the
top level of the unpacked tarball will work on FreeBSD. But the man pages
will be placed under the /usr/local/share/man directory which unfortunately
is not on the standard manpath. One solution is to add this path by
creating a file with a name like local_share.conf in the
/usr/local/etc/man.d/ directory and placing this line in it:
    MANPATH /usr/local/share/man

FreeBSD 9.0 has a "ports" entry for sg3_utils under the
/usr/ports/sysutils directory. It points to version 1.28 of sg3_utils
which is now a bit dated. It could be used as a template to point
to more recent versions.

kFreeBSD
========
sg3_utils can be built into a Debian package for kFreeBSD using the
./build_debian.sh script in the top level directory. This has been tested
with Debian 6.0 release.

Details
=======
Most of the ported utilities listed above use SCSI command functions
declared in sg_cmds_*.h headers . Those SCSI command functions are
implemented in the corresponding ".c" files. The ".c" files pass SCSI
commands to the host operating system via an interface declared in sg_pt.h .
There are currently five implementations of that interface depending on
the host operating system:
  - sg_pt_linux.c
  - sg_pt_freebsd.c
  - sg_pt_osf1.c  [Tru64]
  - sg_pt_win32.c
  - sg_pt_solaris.c

The sg_pt_freebsd.c file uses the FreeBSD CAM SCSI pass through mechanism.
Hence only FreeBSD device nodes that support CAM can be used. These can be
viewed with the "camcontrol devlist" command. To access ATAPI devices (e.g.
ATAPI DVD drives) the kernel may need to be configured with the "atapicam"
device.

Attempts to send SCSI commands with data-in or data-out buffers around 64 KB
and larger failed on a FreeBSD 7.0 with an "argument list too long" error
message. There is an associated kernel message (viewable with dmesg) that an
attempt has been made to map <n> bytes which is greater than
DFLTPHYS(65536). Still a problem in FreeBSD 8.1 . Due to CAM overhead the
largest power of 2 that can fit through with one command is 32768 bytes (32
KB).

FreeBSD 9.0 is the most recent version of FreeBSD tested with these
utilities.



Douglas Gilbert
1st May 2021

iSCSI support for sg3-utils is available from external patches.

To build sg3-utils from sources and activate built-in iSCSI support
you need both sg3-utils and the external user-space iSCSI library hosted at :

https://github.com/sahlberg/libiscsi

This library provides a client library for accessing remote iSCSI
devices and also comes with patches to the sg3-utils source code
distribution to compile a special version of sg3-utils with iSCSI
support.

No support for iSCSI is provided by the sg3-utils maintainer.



Once sg3-utils is compiler and installed with libiscsi support, you
can specify remote iSCSI devices through a special URL format instead
of the normal /dev/* syntax.

Example:

sg_inq iscsi://ronnie%password@10.1.1.27/iqn.ronnie.test/1
standard INQUIRY:
 PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
 [AERC=0]  [TrmTsk=1]  NormACA=0  HiSUP=0  Resp_data_format=2
 SCCS=0  ACC=0  TPGS=0  3PC=0  Protect=0  BQue=0
 EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
 [RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
 [SPI: Clocking=0x0  QAS=0  IUS=0]
   length=66 (0x42)   Peripheral device type: disk
 Vendor identification: IET
 Product identification: VIRTUAL-DISK
 Product revision level: 0001
 Unit serial number:                           beaf11

Hi,

you can use sg_start to start (spin-up, 1) and stop (spin-down, 0) devices.
I also offers a parameter (-s) to send a synchronize cache command to a
device, so it should write back its internal buffers to the medium.

Be aware that the Linux SCSI subsystem at this time does not automatically
starts stopped devices, so stopping a device which is in use may have fatal
results for you.

So, you should apply with care.
I use it in my shutdown script at the end (before the poweroff command):

# SG_SHUG_NOS is set in my config file rc.config
# SG_SHUT_NOS="0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15"
if test -x /bin/sg_start; then
    if test "`basename $command`" = "reboot"; then
        for no in $SG_SHUT_NOS;
	      do /bin/sg_start /dev/sg$no -s >/dev/null 2>&1;
	 done
    else
        for no in $SG_SHUT_NOS;
	    do /bin/sg_start /dev/sg$no -s 0 >/dev/null 2>&1;
        done
    fi
fi

Enjoy!
Kurt Garloff


Postscript
==========
sg_start has been reworked to allow a block device (e.g. /dev/sda) in
addition to the sg device name (e.g. /dev/sg0) in the lk 2.6 series.
sg_start now has more command line options, see its man page.

	Douglas Gilbert <dgilbert at interlog dot com> 2004/5/8

Please Note:
>>> Up to and including sg3_utils-1.33 the Solaris code was built
>>> and tested on an OpenSolaris VM run with VirtualBox on Ubuntu
>>> 11.10 . Now with Ubuntu 12.04 those VMs crash immediately when
>>> started with VirtualBox. Further, Oracle (who owns SUN and thus
>>> Solaris) no longer supports OpenSolaris and its package
>>> repository has been withdrawn. The author can find no generic VMs
>>> for Oracle Solaris 11 that run on VirtualBox or VMWare. The author
>>> is also displeased with the withdrawal of the Open Software
>>> OS and is disinclined to build a Solaris 11 system just to
>>> virtualize it.
>>> So as of sg3_utils-1.34 the Solaris port is provided "as-is" without
>>> testing on a Solaris platform.

Douglas Gilbert
13th October 2012



Introduction
============
The Solaris port of sg3_utils contains those utilities that are
_not_ specific to Linux.

The dd variants from the sg3_utils package (e.g. sg_dd) rely on too many
Linux idiosyncrasies to be easily ported. A new package called 'ddpt'
contains a utility with similar functionality to sg_dd and is available
for Solaris.

Supported Utilities
===================
Here is a list of utilities that have been ported:
    sg_bg_ctl
    sg_compare_and_write
    sg_decode_sense
    sg_format
    sg_get_config
    sg_get_elem_status
    sg_get_lba_status
    sg_ident
    sg_inq          [dropped ATA IDENTIFY DEVICE capability]
    sg_logs
    sg_luns
    sg_modes
    sg_persist
    sg_opcodes
    sg_prevent
    sg_raw
    sg_rdac
    sg_read_block_limts
    sg_read_buffer
    sg_read_long
    sg_readcap
    sg_reassign
    sg_referrals
    sg_rep_pip
    sg_rep_zones
    sg_requests
    sg_rmsn
    sg_rtpg
    sg_safte
    sg_sanitize
    sg_sat_identify
    sg_sat_phy_event
    sg_sat_set_features
    sg_seek
    sg_senddiag
    sg_ses
    sg_start
    sg_stpg
    sg_stream_ctl
    sg_sync
    sg_turs
    sg_unmap
    sg_verify
    sg_vpd
    sg_wr_mode
    sg_write_buffer
    sg_write_long
    sg_write_same
    sg_write_verify
    sg_write_x
    sg_zone

Most utility names are indicative of the main SCSI command
that they execute.  Some utilities are slightly higher level, for
example sg_ses fetches SCSI Enclosure Services (SES) status pages and
can send control pages. Each utility has a man page (placed in
section 8). An overview of sg3_utils can be found at:
https://sg.danny.cz/sg/sg3_utils.html .
A copy of the "sg3_utils.html" file is in the "doc" subdirectory.


The executables and library can be built from the source code in
the tarball and installed with the familiar
"./configure ; make ; make install" sequence. If this fails try
running the "./autogen.sh" script prior to that sequence. There
are generic instruction on configure and friend in the INSTALL file.

Some man pages have examples which use Linux device names which
hopefully will not confuse the Solaris users.

Device naming
=============
In Solaris, SCSI device names below the '/dev' directory have a
form like: c5t4d3s2 where the number following "c" is the controller
(HBA) number, the number following "t" is the target number (from
the SCSI parallel interface days) and the number following "d" is
the LUN. Following the "s" is the slice number which is related to
a partition and by convention "s2" is the whole disk.

OpenSolaris also has a c5t4d3p2 form where the number following
the "p" is the partition number apart from "p0" which is the whole
disk. So a whole disk may be referred to as either:
  - c5t4d3
  - c5t4d3s2
  - c5t4d3p0

And these device names are duplicated in the /dev/dsk and /dev/rdsk
directories. The former is the block device name and the latter
is for "raw" (or char device) access which is what sg3_utils needs.
So in OpenSolaris something of the form:
   sg_inq /dev/rdsk/c5t4d3p0
should work. If it doesn't add a '-vvv' option. If that is attempted
on the /dev/dsk/c5t4d3p0 variant an inappropriate ioctl for device
error will result.

The device names within the /dev directory are typically symbolic
links to much longer topological names in the /device directory.

In Solaris cd/dvd/bd players seem to be treated as disks and so are
found in the /dev/rdsk directory. Tape drives appear in the /dev/rmt
directory.

There is also a sgen (SCSI generic) driver which by default does not
attach to any device. See the /kernel/drv/sgen.conf file to control
what is attached. Any attached device will have a device name of
the form /dev/scsi/c5t4d3 .

Listing available SCSI devices in Solaris seems to be a challenge.
"Use the 'format' command" advice works but seems a very dangerous
way to list devices. [It does prompt again before doing any damage.]
'devfsadm -Cv' cleans out the clutter in the /dev/rdsk directory,
only leaving what is "live". The "cfgadm -v" command looks promising.

Details
=======
The ported utilities listed above, all use SCSI command functions
declared in sg_cmds_basic.h and sg_cmds_extra.h . Those SCSI command
functions are implemented in the corresponding ".c" files. The ".c"
files pass SCSI commands to the host operating system via an interface
declared in sg_pt.h . There are currently five implementations of that
interface depending on the host operating system:
  - sg_pt_linux.c
  - sg_pt_freebsd.c
  - sg_pt_osf1.c  [Tru64]
  - sg_pt_solaris.c
  - sg_pt_win32.c

The sg_pt_solaris.c file uses the "uscsi" SCSI pass through mechanism. There
seems to be no corresponding ATA pass through and recent SATA disks do not
seem to have a SAT layer in front of them (within Solaris). If SAT is
present (perhaps externally or within a HBA) then that would allow SATA
disks to accept SCSI commands including the SCSI ATA PASS THROUGH commands.


Douglas Gilbert
5th June 2020

Introduction
============
The Tru64 port of sg3_utils contains those utilities that are _not_
specific to Linux. In some cases a utility could be ported but
requires more work. An example is sg_dd which needs more work
beyond the SCSI command pass through mechanism.

Supported Utilities
===================
Here is a list of utilities that have been ported:
    sg_compare_and_write
    sg_decode_sense
    sg_format
    sg_get_config
    sg_get_lba_status
    sg_ident
    sg_inq          [dropped ATA IDENTIFY DEVICE capability]
    sg_logs
    sg_luns
    sg_modes
    sg_opcodes
    sg_persist
    sg_prevent
    sg_raw
    sg_rdac
    sg_read_block_limits
    sg_read_buffer
    sg_read_long
    sg_readcap
    sg_reassign
    sg_referrals
    sg_requests
    sg_rmsn
    sg_rtpg
    sg_safte
    sg_sanitize
    sg_sat_identify
    sg_sat_phy_event
    sg_sat_set_features
    sg_senddiag
    sg_ses
    sg_start
    sg_stpg
    sg_sync
    sg_turs
    sg_unmap
    sg_verify
    sg_vpd
    sg_wr_mode
    sg_write_buffer
    sg_write_long
    sg_write_same

Most utility names are indicative of the main SCSI command
that they execute.  Some utilities are slightly higher level, for
example sg_ses fetches SCSI Enclosure Services (SES) status pages and
can send control pages. Each utility has a man page (placed in
section 8). An overview of sg3_utils can be found at:
https://sg.danny.cz/sg/sg3_utils.html .
A copy of the "sg3_utils.html" file is in the "doc" subdirectory.

This package uses autotools infrastructure with the now common
"./configure ; make ; make install" sequence needed to build and install
from the source found in the tarball. If the "./configure" sequence
fails try using the ./autogen.sh prior to that sequence.

Some man pages have examples which use Linux device names which hopefully
will not confuse Tru64 users.


Details
=======
Most of the ported utilities listed above use SCSI command functions
declared in sg_cmds_*.h headers . Those SCSI command functions are
implemented in the corresponding ".c" files. The ".c" files pass SCSI
commands to the host operating system via an interface declared in sg_pt.h .
There are currently five implementations of that interface depending on
the host operating system:
system:
  - sg_pt_linux.c
  - sg_pt_osf1.c  [Tru64]
  - sg_pt_freebsd.c
  - sg_pt_solaris.c
  - sg_pt_win32.c

The sg_pt_osf1.c file uses the Tru64 CAM SCSI pass through mechanism.

Tru64 does not have general library support for "long" options
(e.g. "--verbose") which are used extensively by most of the
utilities in this package. Rather than change all the utilities
and their man/web pages a local implementation of the missing
function "getopt_long()" has been placed in the "getopt_long"
subdirectory. Currently only the Tru64 port uses it.


Douglas Gilbert
14th January 2013

Introduction
============
The win32 port of sg3_utils contains those utilities that are _not_ specific
to Linux. One utility for listing available devices, sg_scan, has a
Windows-specific version for this port.

The dd variants from the sg3_utils package (e.g. sg_dd) rely on too many
Linux idiosyncrasies to be easily ported. A new package called 'ddpt'
contains a utility with similar functionality to sg_dd and is available
for Windows.

The Windows port uses the Microsoft SCSI Pass Through (SPT) interface.
It has two variants: "SPT" where data is double buffered; and "SPTD"
where data pointers to the user space are passed to the OS. Only Windows
2000 and later (i.e. not 95, 98 or ME) support SPT.

Two build environments are catered for: cygwin (see www.cygwin.com) and
MinGW ("Minimalist GNU for Windows", see www.mingw.org). Both are based in
the gcc compiler (although other C compilers should have little problem with
the source code). Cygwin is a more sophisticated, commercial product that
results in executables that depend on cygwin1.dll . No licensing is required
since sg3_utils is open source (with either BSD or GPL licenses) but users
will need to fetch that dll. On the other hand MinGW (and its companion MSYS
shell) builds freestanding console executables. The Unix library support is
not as advanced with MinGW which has led to some timing functions being
compiled out when sg3_utils is built for MinGW.

In later versions of Windows these utilities may need to be "run as
Administrator" for disks and other devices to be seen. If not those devices
will simply not be found as calls to query them fail with access permission
problems.

Supported Utilities
===================
Here is a list of utilities that have been ported:
    sg_bg_ctl
    sg_compare_and_write
    sg_decode_sense
    sg_format
    sg_get_config
    sg_get_elem_status
    sg_get_lba_status
    sg_ident
    sg_inq          [dropped ATA IDENTIFY DEVICE capability]
    sg_logs
    sg_luns
    sg_modes
    sg_opcodes
    sg_persist
    sg_prevent
    sg_raw
    sg_rdac
    sg_read_attr
    sg_read_block_limits
    sg_read_buffer
    sg_read_long
    sg_readcap
    sg_reassign
    sg_referrals
    sg_rep_pip
    sg_rep_zones
    sg_requests
    sg_reset_wp
    sg_rmsn
    sg_rtpg
    sg_safte
    sg_sanitize
    sg_sat_identify
    sg_sat_phy_event
    sg_sat_read_gplog
    sg_sat_set_features
    sg_scan         [this is Windows specific]
    sg_seek
    sg_senddiag
    sg_ses
    sg_ses_microcode
    sg_start
    sg_stpg
    sg_stream_ctl
    sg_sync
    sg_timestamp
    sg_turs
    sg_unmap
    sg_verify
    sg_vpd
    sg_wr_mode
    sg_write_buffer
    sg_write_long
    sg_write_same
    sg_write_verify
    sg_write_x
    sg_zone

Most utility names are indicative of the main SCSI command that they execute.
Some utilities are slightly higher level, for example sg_ses fetches SCSI
Enclosure Services (SES) status pages and can send control pages. Each
utility has a man page (placed in section 8). There is summary of the mapping
between utility names and the SCSI commands they execute in the COVERAGE
file. An overview of sg3_utils can be found at:
https://sg.danny.cz/sg/sg3_utils.html .
A copy of the "sg3_utils.html" file is in the "doc" subdirectory.

Some man pages have examples which use Linux device names which hopefully
will not confuse Windows users.

Two pass-through variants
=========================
The sg_pt_win32.c file uses the Windows SCSI Pass Through interface.
That is often shortened to SPT or SPTI. There are two DeviceIoControl()
ioctl variants provided: IOCTL_SCSI_PASS_THROUGH and
IOCTL_SCSI_PASS_THROUGH_DIRECT. The former involves double handling of
data (and perhaps an upper limit on the data length associated with
one SCSI command; MS documentation mentions 16 KB). The "direct"
variant passes a pointer from the user space and to be faster looks
and more versatile.

However the "direct" variant has potentially (unquantified) alignment
requirements and may not be (well) implemented by the hardware driver.
In practice some users have reported errors (e.g. 1117: a non-descript
IO error) when the direct variant is used.

Hence the non-direct variant is the default. The default size limit
on the data buffer is set at 16 KB but if the user asks for more
the data buffer will be extended. The OS or the hardware drivers
may reject the extended data buffer but we tried.

The package can be built using the direct variant with:
   ./configure --enable-win32-spt-direct
rather than:
   ./configure
prior to the 'make' call.

In sg3_utils version 1.31 run-time selection of the direct or indirect
interface was added with the scsi_pt_win32_direct(int state_direct)
function declared in sg_pt.h. The default is indirect unless
'./configure --enable-win32-spt-direct' was used in the build. If
'state_direct' is 1 then the direct interface is used and if it is 0
the indirect interface is used.

Both sg_read_buffer and sg_write_buffer can transfer buffers larger
than 16 KB. So in sg3_utils version 1.31, they use this new function
to set direct interface mode. This is regardless of whether or
not "--enable-win32-spt-direct" is given to ./configure .

Details
=======
Most of the ported utilities listed above use SCSI command functions
declared in sg_cmds_*.h headers . Those SCSI command functions are
implemented in the corresponding ".c" files. The ".c" files pass SCSI
commands to the host operating system via an interface declared in sg_pt.h .
There are currently five implementations of that interface depending on
the host operating system:
  - sg_pt_linux.c
  - sg_pt_freebsd.c
  - sg_pt_osf1.c  [Tru64]
  - sg_pt_solaris.c
  - sg_pt_win32.c

The ASPI32 interface which requires a dll from Adaptec is not supported.

The sg_scan utility is a special version for Windows and it attempts to show
the various available storage device names, one per line. Here is an example
of sg_scan's output:

# sg_scan
PD0     [C]     FUJITSU   MHY2160BH         0000
PD1     [DF]    WD        2500BEV External  1.05  WD-WXE90
CDROM0  [E]     MATSHITA DVD/CDRW UJDA775  CB03

Here is an example with added bus type:

# sg_scan -b
PD0     [C]     <Ata  >  FUJITSU   MHY2160BH         0000
PD1     [DF]    <Usb  >  WD        2500BEV External  1.05  WD-WXE90
CDROM0  [E]     <Atapi>  MATSHITA DVD/CDRW UJDA775  CB03

Here is an example with added SCSI adapter scan:

# sg_scan -b -s
PD0     [C]     <Ata  >  ST380011A  8.01
PD1             <Scsi >  SEAGATE   ST373455SS        2189
PD2             <Scsi >  ATA       ST3160812AS       D
PD3             <Scsi >  SEAGATE   ST336754SS        0003
CDROM0  [F]     <Atapi>  HL-DT-ST DVDRAM GSA-4163B  A103
TAPE0           <Scsi >  SONY      SDT-7000          0192

SCSI0:0,0,0   claimed=1 pdt=0h dubious  ST380011  A                 8.01
SCSI1:0,0,0   claimed=1 pdt=5h          HL-DT-ST  DVDRAM GSA-4163B  A103
SCSI2:0,6,0   claimed=1 pdt=1h          SONY      SDT-7000          0192
SCSI5:0,17,0  claimed=1 pdt=0h          SEAGATE   ST373455SS        2189
SCSI5:0,19,0  claimed=1 pdt=0h          ATA       ST3160812AS       D
SCSI5:0,21,0  claimed=1 pdt=0h          SEAGATE   ST336754SS        0003
SCSI5:0,112,0 claimed=0 pdt=10h         LSI       PSEUDO DEVICE     2.34

The storage devices scanned are PhysicalDrive<n> (shortened form PD<n> used),
CDROM<n> (which includes DVD and BD drives) and TAPE<n>. There is also an
optional SCSI adapter scan with device names of the form SCSI<n>:<b>:<t>:<l> .
These only come into play for devices that are not claimed by one of the
storage class drivers. The "LSI PSEUDO DEVICE" device above is an example
of an unclaimed device. The SCSI adapter scan does not show USB and IEEE
1394 connected devices.

Volume names (e.g. "C:") that match a storage device (or perhaps a
partition within that device) are shown in brackets. Notice there can be
zero, one or more volume names for each storage device. Up to four volume
names are listed in brackets, if there are more a "+" is added after the
fourth.

Several utilities have conditional compilation sections based on
the SG_LIB_MINGW define. For those who want to try native C compilers
on Windows setting the SG_LIB_MINGW define may help.

Build environments
==================
This package uses autotools infrastructure with the now common
"./configure ; make ; make install" sequence needed to build and install
from the source found in the tarball. Two Windows environments for building
Unix code are supported: cygwin and MinGW. If the "./configure" sequence
fails try using the ./autogen.sh prior to that sequence. The executables
produced are console applications that can be executed in either a cygwin,
MSYS or "cmd" shell. Various build options are available by giving
command line options to "./configure", see the INSTALL file for generic
information about the build infrastructure.

MinGW can be used to cross built on some Redhat (Linux) platforms. After
loading the cross build packages, the ./configure call in the normal
autotools sequence should be replaced by either mingw32-configure or
mingw64-configure. These scripts will set up the environment for
the cross build and then call ./configure (so this invocation should be
made in the top level of the untarred source). Options given to either
script (e.g. --enable-win32-spt-direct) will be passed through to
./configure .

Binary and Text files
=====================
A problem has been reported with binary output being written in a MinGW
environment (or executables build by MinGW). Windows has a concept of text
and binary files which is not found in Unix. Recent versions of MinGW
default to opening files in text mode. This can lead to binary output
(such as when the '--raw' option is given) having 0xa (i.e. LF) translated
to 0xd,0xa (i.e. CR,LF). sg3_utils version 1.26 attempts to fix this
problem by changing what it knows to be binary output files to "binary
mode" with the setmode() Windows command.


Douglas Gilbert
6th June 2020