Skip to content

Overhaul README.md #228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AlexanderRichert-NOAA merged 2 commits into from
Jun 24, 2025
+97 −231
Merged

Overhaul README.md #228

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
56a3592
Overhaul README.md
AlexanderRichert-NOAA Jun 17, 2025
b431e8f
delete build_linux.yml workflow
AlexanderRichert-NOAA Jun 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 0 additions & 51 deletions .github/workflows/build_linux.yml
View file
Open in desktop

This file was deleted.

277 changes: 97 additions & 180 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,54 +2,62 @@

## Introduction

This is a collection of libraries commonly known as NCEPLIBS that are
required for several NCEP applications, e.g., UFS, GSI, UPP, etc. For
general information about NCEPLIBS, the user is referred to the
[Wiki](https://github.com/NOAA-EMC/NCEPLIBS/wiki).

For full documentation of each package see
https://noaa-emc.github.io/NCEPLIBS/.

For installing NCEPLIBS on [NOAA HPC
NCEPLIBS is a collection of libraries used by a range of NCEP R&D and
operational applications, including UFS (Weather Model, SRW, etc.), GSI/Global
Workflow, UFS_UTILS, and UPP. General information about NCEPLIBS, including
code best practices, contributor expectations, and continuous integration
methods, can be found in the [NCEPLIBS
wiki](https://github.com/NOAA-EMC/NCEPLIBS/wiki).

For using NCEPLIBS on [NOAA HPC
systems](https://spack-stack.readthedocs.io/en/latest/PreConfiguredSites.html),
see the
[documentation](https://spack-stack.readthedocs.io/en/latest/Overview.html#getting-started)
for the [spack-stack](https://github.com/JCSDA/spack-stack) project. The
documentation provides instructions for using spack-stack to [build software
stacks from
see the [spack-stack](https://github.com/JCSDA/spack-stack) project's [Getting
Started
documentation](https://spack-stack.readthedocs.io/en/latest/Overview.html#getting-started)
and [wiki](https://github.com/JCSDA/spack-stack/wiki). The spack-stack
framework is used to install a range of libraries, including all currently
supported NCEPLIBS codes, on [RDHPCS](https://docs.rdhpcs.noaa.gov/) systems in
locations available to all users. The documentation also provides instructions
for using spack-stack to [build software stacks from
scratch](https://spack-stack.readthedocs.io/en/latest/CreatingEnvironments.html#creatingenvironment),
as well as to [build stacks that are chained to existing
environments](https://spack-stack.readthedocs.io/en/latest/AddingTestPackages.html#add-test-packages)
in order to re-use dependencies already provided by official spack-stack
deployments. **Note** that users are recommended to use existing spack-stack
installations as much as possible when using NOAA R&D systems, and to use
spack-stack as much as possible when developing for applications that
spack-stack is intended to support, namely, UFS applications, Global Workflow,
and JEDI applications (i.e., users developing the UFS Weather Model on a
personal machine are recommended to use their own installation of spack-stack).
Outside of these platforms and applications, users are recommended to install
NCEPLIBS either through CMake or through regular Spack (not spack-stack); see
instructions below.

NCEPLIBS is comprised of the following libraries:
deployments.

> [!WARNING]
> This repository contains long disused scripts for compiling NCEPLIBS. **These
Comment thread
AlexanderRichert-NOAA marked this conversation as resolved.
are not in any way maintained or supported. Use as at your own risk.** See
instructions below for obtaining NCEPLIBS.

NCEPLIBS is comprised of the following libraries. *Italicization* indicates a
repository that as of Summer 2025 is no longer actively developed, therefore
NOAA developers are suggested to avoid using them, or to confer with the NCEPLIBS
team before beginning any development work with them.

NCEP library | Spack package name | Description
----------------------|--------------------|-----------------------------------------------------
[NCEPLIBS-bacio](https://github.com/NOAA-EMC/NCEPLIBS-bacio) | bacio | Binary I/O for the NCEP models
[NCEPLIBS-bufr](https://github.com/NOAA-EMC/NCEPLIBS-bufr) | bufr | Working with the BUFR format
[NCEPLIBS-g2](https://github.com/NOAA-EMC/NCEPLIBS-g2) | g2 | Coding/decoding GRIB2 messages
[NCEPLIBS-g2c](https://github.com/NOAA-EMC/NCEPLIBS-g2c) | g2c | C decoder/encoder routines for GRIB edition 2
[NCEPLIBS-g2tmpl](https://github.com/NOAA-EMC/NCEPLIBS-g2tmpl) | g2tmpl | Utilities for GRIB2 templates
[NCEPLIBS-gfsio](https://github.com/NOAA-EMC/NCEPLIBS-gfsio) | gfsio | Convert GFS Gaussian output into grib output
[NCEPLIBS-ip](https://github.com/NOAA-EMC/NCEPLIBS-ip) | ip | General interpolation library
[NCEPLIBS-landsfcutil](https://github.com/NOAA-EMC/NCEPLIBS-landsfcutil) | landsfcutil | Initializing land-surface states
[NCEPLIBS-ncio](https://github.com/NOAA-EMC/NCEPLIBS-ncio) | ncio | Read the GFS forecast files
[NCEPLIBS-nemsio](https://github.com/NOAA-EMC/NCEPLIBS-nemsio) | nemsio | I/O for the NCEP models using NEMS
[NCEPLIBS-nemsiogfs](https://github.com/NOAA-EMC/NCEPLIBS-nemsiogfs) | nemsiogfs | I/O for the NEMS-GFS model
[NCEPLIBS-sfcio](https://github.com/NOAA-EMC/NCEPLIBS-sfcio) | sfcio | Surface files I/O
[NCEPLIBS-sigio](https://github.com/NOAA-EMC/NCEPLIBS-sigio) | sigio | I/O on the sigma restart file of the global spectral model
[NCEPLIBS-w3emc](https://github.com/NOAA-EMC/NCEPLIBS-w3emc) | w3emc | Decoder/encoder routines for GRIB edition 1
[NCEPLIBS-wrf_io](https://github.com/NOAA-EMC/NCEPLIBS-wrf_io) | wrf-io | Lightweight WRF-IO API library for Unified Post Processor (UPP)
[NCEPLIBS-bacio](https://github.com/NOAA-EMC/NCEPLIBS-bacio) | bacio | Binary I/O for the NCEP models
[NCEPLIBS-bufr](https://github.com/NOAA-EMC/NCEPLIBS-bufr) | bufr | Working with the BUFR format
[NCEPLIBS-g2](https://github.com/NOAA-EMC/NCEPLIBS-g2) | g2 | Coding/decoding GRIB2 messages
[NCEPLIBS-g2c](https://github.com/NOAA-EMC/NCEPLIBS-g2c) | g2c | C decoder/encoder routines for GRIB edition 2
[NCEPLIBS-g2tmpl](https://github.com/NOAA-EMC/NCEPLIBS-g2tmpl) | g2tmpl | Utilities for GRIB2 templates
*[NCEPLIBS-gfsio](https://github.com/NOAA-EMC/NCEPLIBS-gfsio)* | gfsio | Convert GFS Gaussian output into grib output
*[NCEPLIBS-grib_util](https://github.com/NOAA-EMC/NCEPLIBS-grib_util)* | grib-util | Utilities for working with GRIB2 files
[NCEPLIBS-ip](https://github.com/NOAA-EMC/NCEPLIBS-ip) | ip | General interpolation library
*[NCEPLIBS-landsfcutil](https://github.com/NOAA-EMC/NCEPLIBS-landsfcutil)* | landsfcutil | Initializing land-surface states
[NCEPLIBS-ncio](https://github.com/NOAA-EMC/NCEPLIBS-ncio) | ncio | Read the GFS forecast files
*[NCEPLIBS-nemsio](https://github.com/NOAA-EMC/NCEPLIBS-nemsio)* | nemsio | I/O for the NCEP models using NEMS
*[NCEPLIBS-nemsiogfs](https://github.com/NOAA-EMC/NCEPLIBS-nemsiogfs)* | nemsiogfs | I/O for the NEMS-GFS model
*[NCEPLIBS-prod_util](https://github.com/NOAA-EMC/NCEPLIBS-prod_util)* | prod-util | EMC copy of NCO's utilities for misc. date & file manipulation
*[NCEPLIBS-sfcio](https://github.com/NOAA-EMC/NCEPLIBS-sfcio)* | sfcio | Surface files I/O
*[NCEPLIBS-sigio](https://github.com/NOAA-EMC/NCEPLIBS-sigio)* | sigio | I/O on the sigma restart file of the global spectral model
[NCEPLIBS-w3emc](https://github.com/NOAA-EMC/NCEPLIBS-w3emc) | w3emc | Decoder/encoder routines for GRIB edition 1
*[NCEPLIBS-wrf_io](https://github.com/NOAA-EMC/NCEPLIBS-wrf_io)* | wrf-io | Lightweight WRF-IO API library for Unified Post Processor (UPP)
[wgrib2](https://github.com/NOAA-EMC/wgrib2) | wgrib2 | Utility for working with GRIB2 files

For information on maintainers, documentation, and bug and feature requests,
refer to each package repository listed above.

The [NCEPLIBS-w3nco](https://github.com/NOAA-EMC/NCEPLIBS-w3nco),
[NCEPLIBS-ip2](https://github.com/NOAA-EMC/NCEPLIBS-ip2),
Expand All @@ -58,60 +66,37 @@ and [NCEPLIBS-sp](https://github.com/NOAA-EMC/NCEPLIBS-sp) libraries
NCEPLIBS-w3emc replaces NCEPLIBS-w3nco, and NCEPLIBS-ip replaces NCEPLIBS-ip2
and NCEPLIBS-sp.

The following projects are also related to the NCEPLIBS project:
- [NCEPLIBS-grib_util](https://github.com/NOAA-EMC/NCEPLIBS-grib_util) ('grib-util' in Spack)
- [NCEPLIBS-prod_util](https://github.com/NOAA-EMC/NCEPLIBS-prod_util) ('prod-util' in Spack)
- [UFS_UTILS](https://github.com/NOAA-EMC/UFS_UTILS) ('ufs-utils' in Spack)

## Required Software

- A supported C/C++ and Fortran compiler (see table below). Other
versions may work, in particular if close to the versions listed
## Supported compilers and required software

NCEPLIBS codes are tested with and intended to support GCC, Intel Classic, and
Intel oneAPI compilers. Older versions of each of these may not be compatible.
Refer to individual code repositories (including GitHub Issues) for more
specific information on compiler conflicts, and please submit issues if
previously unknown conflicts emerge. Contributions of modified code and CMake
configurations to allow for the use of other compilers are welcome, but
compilers other than those listed above are not officially supported.

All NCEPLIBS codes use the CMake build system, therefore CMake and at least one
of GMake and Ninja are necessary to build them.

## Obtaining, building, and using NCEPLIBS

> [!NOTE]
> Users are recommended to use existing spack-stack installations as much
as possible when using NOAA R&D systems, and to use spack-stack as much as
possible when developing the applications that spack-stack is intended to
support, namely, UFS applications, Global Workflow, and JEDI applications
(for instance, users developing the UFS Weather Model on a personal machine are
recommended to build a local copy of spack-stack). Outside of these
platforms and applications, users are recommended to install NCEPLIBS either
through CMake or through regular Spack (not spack-stack); see instructions
below.

| Compiler vendor | Supported (tested) versions |
|-----------------|------------------------------------------------------------|
| Intel | 18.0.3.222, 18.0.5.274, 19.0.2.187, 19.0.5.281, 19.1.0.166 |
| GNU | 8.3.0, 9.X.0 |

- A supported MPI library (see table below). Other versions may work,
in particular if close to the versions listed below.

| MPI library | Supported (tested) versions |
|-----------------|------------------------------------------------------------|
| MPICH | 3.3.1, 3.3.2 |
| MVAPICH2 | 2.3.3 |
| Open MPI | 3.1.5, 4.0.2 |
| Intel MPI | 2018.0.4, 2019.6.154, 2020.0.166 |
| SGI MPT | 2.19 |

- A software stack consisting of the following third-party libraries,
compiled with the same compiler and MPI library (where
applicable). Other versions may work, in particular if close to the
versions listed below. To install the third-party libraries needed by
NCEPLIBS, see
[NCEPLIBS-external](https://github.com/NOAA-EMC/NCEPLIBS-external).

| Library | Supported (tested) versions |
|-----------------|------------------------------------------------------------|
| CMake | cmake-3.16.3 |
| zlib | zlib-1.2.11 |
| libpng | libpng-1.6.35 |
| libjpeg | jpeg-9.1 |
| Jasper | jasper-2.0.16 |
| HDF5 | hdf5-1.10.4 |
| NetCDF | netcdf-c-4.7.3, netcdf-fortran-4.5.2 |

The [NCEPLIBS-external](https://github.com/NOAA-EMC/NCEPLIBS-external)
project provides an convenient way to install the required third-party libraries
on supported platforms such as generic Linux/macOS systems.

## Obtaining, Building, and Using NCEPLIBS

### With Spack
All of the NCEPLIBS can be installed through the [Spack](https://spack.io/)

All of the NCEPLIBS codes can be installed through the [Spack](https://spack.io/)
package manager. One significant benefit of using Spack is that it will
automatically install any dependencies needed, including CMake, or even
automatically install any dependencies needed, including CMake, and even
compilers. See the [Spack
documentation](https://spack.readthedocs.io/en/latest/basic_usage.html) for
details of configuring and executing Spack installations. Spack also has
Expand All @@ -121,113 +106,45 @@ and creating Docker and Singularity container images.

In the simplest case, with the `spack` command loaded into the current
environment, run, for example, `spack install ip` to install the latest release
of the NCEPLIBS-ip library along with any needed dependencies. See the above
table for each package's name in Spack. The develop branch for each library can
be installed by specifying the 'develop' version, i.e., `spack install
w3emc@develop`.
of the NCEPLIBS-ip library; Spack will take care of installing all necessary
dependencies. See the above table for each package's name in Spack. The develop
branch for each library can be installed by specifying the 'develop' version,
i.e., `spack install w3emc@develop`.

Here is a typical set of steps for installing and using NCEPLIBS using Spack,
assuming that Spack's
assuming that Spack's own
[prerequisites](https://spack.readthedocs.io/en/latest/getting_started.html#system-prerequisites)
are already installed:
```
git clone https://github.com/spack/spack
. spack/share/spack/setup-env.sh # assuming bash, zsh, or dash shell
spack external find cmake gmake # find external packages to save time
spack install ip@develop precision=4 ^sp@2.4.0 precision=4
spack external find cmake gmake openblas # optionally find external packages to save time
spack install ip@develop precision=4,d,8
spack load ip
```
The `spack load` command configures the $CMAKE_PREFIX_PATH variable,
in this example making the ip library available to downstream CMake builds.
The `spack load` command configures the `$CMAKE_PREFIX_PATH` variable, in this
example making the ip library available to downstream CMake projects (e.g., UFS
Weather Model).

For any bugs or other issues with Spack-based installations, such as incorrect
dependencies in a package's recipe, please file an issue under the appropriate
NCEPLIBS repository.
dependencies in a package's recipe, please file an issue under the main Spack
repository.

### With CMake
#### External dependencies
See [NCEPLIBS-external](https://github.com/NOAA-EMC/NCEPLIBS-external) for
building external dependencies.

#### Obtaining the code

Clone the repository:

`git clone https://github.com/NOAA-EMC/NCEPLIBS`

#### Building the libraries

`CMake` employs an out-of-source build. Create a directory for
configuring the build and cd into it:

`mkdir -p build && cd build`

Set the compilers, if needed, to match those being used for compiling
the third-party libraries listed above: `FC`, `CC`, and `CXX` environment variables can
be used to point to the desired Fortran, C, and C++ compilers.

Execute `cmake` from inside your build directory.

`cmake -DCMAKE_INSTALL_PREFIX=<nceplibs-prefix> ../NCEPLIBS`

If the third-party libraries are not located in a path recognized by `cmake`
e.g. `/usr/local`, it may be necessary to provide the appropriate
environment variables e.g. `<package_ROOT>` or `CMAKE_PREFIX_PATH` so
that `cmake` is able to locate these dependencies.

The installation prefix for NCEPLIBS is provided by the `cmake`
command-line argument `-DCMAKE_INSTALL_PREFIX=<nceplibs-prefix>`

To build and install:

```
make -j<x>
```

This will build and install all the NCEPLIBS and place them into the
installation prefix `<nceplibs-prefix>`.

Additional `cmake` command-line options while configuring NCEPLIBS:

`-DFLAT=ON|OFF` - determines the installation tree heirarchy

`-DDEPLOY=ON|OFF` - prepares and deploys Lua modulefiles for each of
the NCEPLIBS libraries. To deploy modulefiles:

```
make deploy
```

[LMod](https://lmod.readthedocs.io/en/latest/) Environment module
system is required to load/unload these modules.

Some HPC systems use Tcl based Environment modules. NCEPLIBS provides
a means to deploy Tcl module files and can be exercised as follows:

`-DTCLMOD=ON` - enables option to change Lua modulefiles to Tcl modulefiles for each of the NCEPLIBS libraries. To make the change:
```
make tcl
```

On machines that do not provide access to Github, a `DOWNLOAD_ONLY`
and `USE_LOCAL` option is provided. To download the NCEPLIBS without
building them use the cmake command line option `-DDOWNLOAD_ONLY=ON`.
This will download all the NCEPLIBS in a local `download` folder.
`make tarball` will tar and zip this folder into `nceplibs.tgz` which
can then be transferred to the machine without GitHub access. Untar
the tarball, and use the cmake command line option `USE_LOCAL` as
`-DUSE_LOCAL=ON`. This will use the source code from the download
directory.
For CMake build instructions, including software prerequisities and build
options, refer to each NCEPLIBS repository's README and ReadTheDocs page. The
GitHub Actions workflows under .github/workflows/ in each NCEPLIBS repository
may be useful references for possible approaches to obtaining software prerequisites.

## Usage

NCEPLIBS can be used in any application that uses `cmake` to
configure and build by adding `-DCMAKE_PREFIX_PATH=<nceplibs-prefix>`
to the cmake command line during configuration. Note that if NCEPLIBS
are installed through Spack, loading the packages, either through
Spack-built environment modules or using the `spack load` command,
will populate the `$CMAKE_PREFIX_PATH` environment variable with
the appropriate paths.
NCEPLIBS can be used in any application that uses `cmake` to configure and
build by adding `-DCMAKE_PREFIX_PATH=<nceplibs-prefix>` to the CMake command
line during configuration. Note that if NCEPLIBS codes are installed through
Spack, loading the packages, either through Spack-built environment modules or
using the `spack load` command, will by default populate the
`$CMAKE_PREFIX_PATH` environment variable with the appropriate paths.

## Disclaimer

Expand Down