[virt-tools-list] [Libguestfs] ANNOUNCE: libnbd 1.2 & nbdkit 1.16 - high performance NBD client and server

Brett Thurber bthurber at redhat.com
Thu Nov 14 19:17:32 UTC 2019


Very nice and congrats on getting this in.

Brett

On Thu, Nov 14, 2019 at 5:53 AM Richard W.M. Jones <rjones at redhat.com>
wrote:

> I'm pleased to announce the releases of libnbd 1.2 and nbdkit 1.16.
> These are a high performance Network Block Device (NBD) client library
> and server.
>
>
> Key features of libnbd:
>
>  * Synchronous API for ease of use.
>  * Asynchronous API for writing non-blocking, multithreaded clients.
>    You can mix both APIs freely.
>  * High performance.
>  * Minimal dependencies for the basic library.
>  * Well-documented, stable API.
>  * Bindings in several programming languages.
>  * Shell (nbdsh) for command line and scripting.
>
> Git: https://github.com/libguestfs/libnbd
> Download: http://download.libguestfs.org/libnbd/1.2-stable/
> Fedora: https://koji.fedoraproject.org/koji/packageinfo?packageID=28807
>
>
> Key features of nbdkit:
>
>  * Multithreaded NBD server written in C with good performance.
>  * Minimal dependencies for the basic server.
>  * Liberal license (BSD) allows nbdkit to be linked to proprietary
>    libraries or included in proprietary code.
>  * Well-documented, simple plugin API with a stable ABI guarantee.
>    Lets you export “unconventional” block devices easily.
>  * You can write plugins in C, Lua, Perl, Python, OCaml, Ruby, Rust,
>    shell script or Tcl.
>  * Filters can be stacked in front of plugins to transform the output.
>
> Git: https://github.com/libguestfs/nbdkit
> Download: http://download.libguestfs.org/nbdkit/1.16-stable/
> Fedora: https://koji.fedoraproject.org/koji/packageinfo?packageID=16469
>
>
> *** Release notes for libnbd 1.2 ***
>
>        These are the release notes for libnbd stable release 1.2.  This
>        describes the major changes since 1.0.
>
>        libnbd 1.2.0 was released on 14th November 2019.
>
>    Security
>        Two security problems were found during development of libnbd 1.2.
>        Both were backported to the 1.0 stable branch.  Upgrading is highly
>        advisable.
>
>        CVE-2019-14842 protocol downgrade attack when using
>        "LIBNBD_TLS_REQUIRE"
>
>        See the full announcement and links to mitigation, tests and fixes
>        here:
>
> https://www.redhat.com/archives/libguestfs/2019-September/msg00128.html
>
>        remote code execution vulnerability
>
>        See the full announcement here:
>
> https://www.redhat.com/archives/libguestfs/2019-October/msg00060.html
>
>    New APIs
>        nbd_can_fast_zero(3)
>            Test support by the server for fast zeroing (Eric Blake).
>
>        nbd_connect_socket(3)
>        nbd_aio_connect_socket(3)
>            Connect to a local connected socket which you create in your
> main
>            program using your own chosen method.
>
>        nbd_connect_systemd_socket_activation(3)
>        nbd_aio_connect_systemd_socket_activation(3)
>            Connect to local processes that support systemd socket
> activation.
>
>        nbd_connect_vsock(3)
>        nbd_aio_connect_vsock(3)
>            Used to connect to servers over "AF_VSOCK".
>
>        nbd_get_handshake_flags(3)
>        nbd_set_handshake_flags(3)
>        nbd_get_request_structured_replies(3)
>        nbd_set_request_structured_replies(3)
>        nbd_get_structured_replies_negotiated(3)
>            Can be used when testing NBD servers to avoid various NBD
> features
>            (Eric Blake).
>
>        nbd_get_protocol(3)
>            Get the NBD protocol variant that the server supports.
>
>        nbd_get_tls_negotiated(3)
>            Did we actually negotiate a TLS connection?
>
>        nbd_set_uri_allow_local_file(3)
>        nbd_set_uri_allow_tls(3)
>        nbd_set_uri_allow_transports(3)
>            These can be used to filter NBD URIs before calling
>            nbd_connect_uri(3).
>
>    New features
>        New tool nbdfuse(1) lets you create a loop-mounted file backed by an
>        NBD server without needing root.
>
>        "AF_VSOCK" is now a supported protocol (thanks Stefan Hajnoczi and
>        Stefano Garzarella).
>
>        Support for the "FAST_ZERO" flag (Eric Blake).
>
>        Allow disabling certain protocol features, to make it easier to test
>        servers (Eric Blake).
>
>        Stack-allocated Variable Length Arrays (VLAs) are now banned
> throughout
>        the library, making the library easier to consume from threads and
>        other small stack situations.
>
>        Reproducible builds (Chris Lamb).
>
>        Support for filtering potentially dangerous or undesirable NBD URI
>        features.
>
>    Documentation
>        Many improvements to the generated manual pages, including:
>
>        ·   Separate "RETURN VALUE" and "ERRORS" sections for each API
>            function.
>
>        ·   Example code.
>
>        ·   Relevant links can be added to the "SEE ALSO" section.
>
>        ·   Link to NBD URI specification where relevant, and improve
>            documentation around what URIs libnbd supports.
>
>        ·   Document libnbd version number scheme.
>
>        ·   Document limits on export name length, encoding etc.
>
>        New libnbd-security(3) man page listing past security issues and
>        remediations (Eric Blake).
>
>    Tools
>        nbdsh(1) has a new --base-allocation option which can be used to
>        request "base:allocation" metadata context.
>
>        New nbdsh(1) --uri (-u) option to connect to URIs.
>
>    Tests
>        You can now fuzz libnbd using either American Fuzzy Lop or clang’s
>        libFuzzer.
>
>        Add unit tests for nbdsh(1) (Eric Blake).
>
>        Improved interop testing with various NBD servers and features.
>
>    Other improvements and bug fixes
>        nbd_connect_tcp(3) now tries to return the correct errno(3) from the
>        underlying connect(2) call when that fails.
>
>        The nbd-protocol.h header file is now shared between libnbd and
> nbdkit.
>
>        Better fork-safety in "nbd_connect_*" APIs.
>
>        The code was analyzed with Coverity and various problems identified
> and
>        fixed.
>
>
> *** Release notes for nbdkit 1.16 ***
>
>        These are the release notes for nbdkit stable release 1.16.  This
>        describes the major changes since 1.14.
>
>        nbdkit 1.16.0 was released on 14th November 2019.
>
>    Security
>        Two security issues were found during development of nbdkit 1.16.
>        Fixes for these were backported to older stable branches.
> Upgrading to
>        the fixed versions is highly recommended.  The new
> nbdkit-security(1)
>        man page contains an up to date list of past security issues.
>
>        CVE-2019-14850 denial of service due to premature opening of
> back-end
>        connection
>
>        See the full announcement and links to mitigation, tests and fixes
>        here:
>
> https://www.redhat.com/archives/libguestfs/2019-September/msg00084.html
>
>        CVE-2019-14851 assertion failure by issuing commands in the wrong
> order
>
>        This CVE was caused by the fix to the previous issue.
>
>        See the full announcement and links to mitigation, tests and fixes
>        here:
>
> https://www.redhat.com/archives/libguestfs/2019-September/msg00272.html
>
>    New features
>        Add support for fast zeroing.  Plugins can expose this using the new
>        ".can_fast_zero" method (Eric Blake).
>
>        nbdkit-partitioning-plugin(1) allows use of "mbr-id=default" or
>        "type-guid=default" to go back to the default MBR byte or partition
>        type GUID.
>
>        New --mask-handshake server flag can be used for testing client
> feature
>        negotiation (Eric Blake).
>
>        The client export name is passed to nbdkit-captive(1) --run
> parameter
>        as $exportname (Eric Blake).
>
>        Captive --run commands which fail (eg. aborting) now cause nbdkit to
>        exit with an error instead of errors being silently ignored (Eric
>        Blake).
>
>        File descriptors can be passed to password parameters, eg:
>        "password=-3" which means that the password should be read from file
>        descriptor 3.
>
>        nbdkit can now serve over the "AF_VSOCK" protocol (thanks Stefan
>        Hajnoczi).
>
>        New --log=null option discards error messages.
>
>    Plugins
>        Python 2 support has been dropped from nbdkit-python-plugin(3) in
> line
>        with Python 2 end of life at the beginning of 2020.  Python ≥ 3.3 is
>        required by this plugin.  If you wish to continue to use Python 2
> then
>        you will need to use nbdkit 1.14.
>
>        New nbdkit-info-plugin(1) which returns various server information
> back
>        to the client.  It can be used for testing server latency amongst
> other
>        things.
>
>        nbdkit-data-plugin(1) now allows you to write "BYTE*N" to get
> repeated
>        bytes (eg. nbdkit data data="0x55*4096").
>
>        nbdkit-ssh-plugin(1) new parameter "compression=true|false" to
> control
>        transport compression.
>
>        nbdkit-vddk-plugin(1) is no longer compiled on non-x86 platforms
> since
>        VMware has only ever shipped VDDK on x86.
>
>        nbdkit-sh-plugin(1) scripts can now see the client exportname and
> can
>        use the "magic_config_key" feature.
>
>    Filters
>        New nbdkit-retry-filter(1) which can reopen the plugin
> transparently on
>        certain types of failures (lots of help from Eric Blake).
>
>    API
>        Macros "NBDKIT_VERSION_MAJOR", "NBDKIT_VERSION_MINOR",
>        "NBDKIT_VERSION_MICRO" expose the compile-time version of nbdkit to
>        plugins and filters (Eric Blake).
>
>        Filters (which unlike plugins do not have a public stable API) must
> now
>        exactly match the version of nbdkit when loaded (Eric Blake).
>
>        New ".can_fast_zero" method (Eric Blake).
>
>        New "nbdkit_export_name" server function for reading the export name
>        passed by the client.
>
>        New "nbdkit_peer_name" server function to return the client address
>        (like getpeername(2)).
>
>        New server functions for safely parsing integers:
> "nbdkit_parse_int",
>        "nbdkit_parse_unsigned", "nbdkit_parse_int8_t",
> "nbdkit_parse_uint8_t",
>        "nbdkit_parse_int16_t", "nbdkit_parse_uint16_t",
>        "nbdkit_parse_int32_t", "nbdkit_parse_uint32_t",
>        "nbdkit_parse_int64_t", "nbdkit_parse_uint64_t".
>
>    Bug fixes
>        ".trim" with FUA flag set now works (Eric Blake).
>
>    Documentation
>        The previous release notes have been turned into man pages.
>
>    Tests
>        Several tests now optionally use nbdsh(1) instead of qemu-io.
>
>        You can now fuzz nbdkit using either American Fuzzy Lop or clang’s
>        libFuzzer.
>
>        Several tests have had sleep times increased to make them more
> stable
>        when run on slow or heavily loaded machines.
>
>    Internals
>        Reproducible builds (Chris Lamb).
>
>        Compile code with -Wshadow warning (Eric Blake).
>
>        The internal backend system has been extensively overhauled.  In
>        particular this means that we now validate request ranges as
> requests
>        are passed between filters and down to the plugin, making it easier
> to
>        find bugs in filters early (Eric Blake).
>
>        Plugin size and "can_*" flags are cached more aggressively by the
>        server (Eric Blake).
>
>        Variable Length Arrays (VLAs) on stack are now banned throughout the
>        code.
>
>        The nbd-protocol.h header describing the NBD protocol is now shared
>        with libnbd(3).
>
>        Plugin ".unload" method is now called after all worker threads have
>        exited, avoiding races at server shutdown.
>
>        Code was audited using Coverity and various problems were fixed.
>
>
>
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-p2v converts physical machines to virtual machines.  Boot with a
> live CD or over the network (PXE) and turn machines into KVM guests.
> http://libguestfs.org/virt-v2v
>
> _______________________________________________
> Libguestfs mailing list
> Libguestfs at redhat.com
> https://www.redhat.com/mailman/listinfo/libguestfs



-- 
Brett Thurber - RHCA, RHCVA
Distinguished Engineer and Engineering Manager, Migration Engineering
Products & Technologies Group, Red Hat
Mobile: +1 (512) 547-9282
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/virt-tools-list/attachments/20191114/1aabe367/attachment.htm>


More information about the virt-tools-list mailing list