7. CCPP Code Management¶

7.1. Organization of the Code¶

This chapter describes the organization of the code, provides instruction on the GitHub workflow and the code review process, and outlines the release procedure. It is assumed that the reader is familiar with using basic GitHub features. A GitHub account is necessary if a user would like to make and contribute code changes.

7.1.1. UFS Atmosphere¶

The UFS Atmosphere source code is contained in the ufs-weather-model code repository. CCPP users and developers should use the ufs-weather-model code and the associated submodules maintained in GitHub. The umbrella repository is located at:

https://github.com/ufs-community/ufs-weather-model

There are nine submodules referenced in the ufs-weather-model repository:

https://github.com/NCAR/ccpp-framework

https://github.com/NCAR/ccpp-physics

https://github.com/NOAA-EMC/fv3atm

https://github.com/NOAA-EMC/GFDL_atmos_cubed_sphere

https://github.com/NOAA-EMC/NEMS

https://github.com/NOAA-EMC/FMS

https://github.com/noaa-psd/stochastic_physics

https://github.com/NOAA-EMC/WW3

https://github.com/NOAA-EMC/NCEPLIBS-pyprodutil

Users have read-only access to these repositories and as such cannot accidentally destroy any important (shared) branches of these authoritative repositories.

All of these repositories are public (no GitHub account required) and may be used directly to read or create forks. Write permission is generally restricted, however. The following table shows the default branch of each authoritative repository and are recommended for CCPP users and developers:

Repository (ufs-weather-model release version)	Branch name	Directory location under ufs-weather-model
https://github.com/ufs-community/ufs-weather-model	develop	.
https://github.com/NCAR/ccpp-physics	master	FV3/ccpp/physics
https://github.com/NCAR/ccpp-framework	master	FV3/ccpp/framwwork
https://github.com/NOAA-EMC/fv3atm	develop	FV3
https://github.com/NOAA-EMC/GFDL_atmos_cubed_sphere	dev/emc	FV3/atmos_cubed_sphere
https://github.com/NCAR/NEMS	develop	NEMS
https://github.com/NCAR/FMS	master	FMS
https://github.com/noaa-psd/stochastic_physics	master	stochastic_physics
https://github.com/NOAA-EMC/WW3	master	WW3
https://github.com/NOAA-EMC/NCEPLIBS-pyprodutil	develop	NEMS/tests/produtil/NCEPLIBS-pyprodutil

7.1.2. SCM¶

CCPP developers should use the SCM code and submodules maintained by GMTB in GitHub. The ccpp-framework and ccpp-physics repositories are used by the two models, creating a two-way dependency between them (see Committing Changes to your Fork for implications on the code review and testing procedure).

https://github.com/NCAR/gmtb-scm

As with ufs-weather-model, there are two submodules referenced in the gmtb-scm repository:

https://github.com/NCAR/ccpp-framework

https://github.com/NCAR/ccpp-physics

Users have read-only access to these repositories and as such cannot accidentally destroy any important (shared) branches of these authoritative repositories. Both CCPP repositories and the SCM repositories are public (no GitHub account required) and may be used directly to read or create forks. Write permission is generally restricted, however.

The following branches are recommended for CCPP users and developers:

Repository (GMTB development version)	Branch name
https://github.com/NCAR/gmtb-scm	master
https://github.com/NCAR/ccpp-physics	master
https://github.com/NCAR/ccpp-framework	master

7.1.3. Directory Structure of ccpp/framework¶

The following is the directory structure for the ccpp/framework (condensed version):

├── cmake                  # cmake files for building
├── doc                    # Documentation for design/implementation and developers guide
│   ├── DevelopersGuide
│   │   └── images
│   └── img
├── schemes                # Example ccpp_prebuild_config.py
│   ├── check
├── scripts                # Scripts for ccpp_prebuild.py, metadata parser, etc.
│   ├── fortran_tools
│   └── parse_tools
├── src                    # CCPP framework source code
│   └── tests              # SDFs and code for testing
├── test
│   └── nemsfv3gfs         # NEMSfv3gfs regression test scripts
└── tests                  # Development for framework upgrades

7.1.4. Directory Structure of ccpp/physics¶

The following is the directory structure for the ccpp/physics (condensed version):

├── physics                 # CCPP physics source code and metadata files
│   ├── docs                # Scientific documentation (doxygen)
│   │   ├── img             # Figures for doxygen
│   │   └── pdftxt          # Text files for documentation

7.2. GitHub Workflow (setting up development repositories)¶

The CCPP development practices make use of the GitHub forking workflow. For users not familiar with this concept, this website provides some background information and a tutorial.

7.2.1. Creating Forks¶

The GitHub forking workflow relies on forks (personal copies) of the shared repositories on GitHub. These forks need to be created only once, and only for directories that users will contribute changes to. The following steps describe how to create a fork for the example of the ccpp-physics submodule/repository:

Go to https://github.com/NCAR/ccpp-physics and make sure you are signed in as your GitHub user.

Select the “fork” button in the upper right corner.

If you have already created a fork, this will take you to your fork.

If you have not yet created a fork, this will create one for you.

Note that the repo name in the upper left (blue) will be either “NCAR” or “your GitHub name” which tells you which fork you are looking at.

Note that personal forks are not required until a user wishes to make code contributions. The procedure for how to check out the code laid out below can be followed without having created any forks beforehand.

7.2.2. Checking out the Code (ufs-weather-model)¶

Start with checking out the main repository from the ufs-community GitHub:

git clone https://github.com/ufs-community/ufs-weather-model
cd ufs-weather-model
git checkout ufs_public_release
git submodule update --init --recursive

At this point, you have checked out the correct branches of all tem repositories. Each repository is connected to a remote destination (simply called remote in GitHub language), which points to the ufs-community GitHub pages and is labeled as origin. For consistency with the CCPP workflow, we recommend renaming the ufs-community remote destination to upstream for any repositories that you will be modifying in order to guarantee that your development will be pushed to your fork and not directly to the ufs-community repository. For example:

cd FV3/ccpp/framework
git remote rename origin upstream
cd ../../..

Checking out remote branches as submodules means that your local branches are in a detached head state, since the submodule points to a specific commit. As long as you are not making any modifications in a certain repository, this is not a problem. If during your development changes are made to the corresponding upstream branch, you can simply navigate to this repository and check out the updated version (example NEMS):

cd NEMS
git remote update
git remote rename origin upstream
git checkout upstream/ufs_public_release
cd ..

However, if you are making changes in a repository (submodule or main repository), you must create a local branch, for example in ufs-weather-model:

git checkout -b my_local_development_branch

Once you are ready to contribute the code to the upstream repository, you need to create a pull request (PR; see next section). In order to do so, you will use your own fork of this repository (see previous section) and configure your fork as an additional remote destination, which we typically label as origin. For the same example ufs-weather-model:

git remote add origin https://github.com/YOUR_GITHUB_USER/ufs-weather-model
git remote update

Then, push your local branch to your fork:

git push origin my_local_development_branch

For each repository/submodule, you can check the configured remote destinations and all existing branches (remote and local):

git remote -v show
git remote update
git branch -a

As opposed to branches without modifications described in step 3, changes to the upstream repository can be brought into the local branch by pulling them down. For the example of submodule FV3 (where a local branch is checked out):

cd FV3
git remote update
git pull upstream ufs_public_release

7.2.3. Checking out the Code (SCM)¶

The process for checking out SCM is described in the following, assuming access via https rather than ssh. We strongly recommend setting up passwordless access to GitHub (see https://help.github.com/categories/authenticating-to-github).

Start with checking out the main repository from the NCAR GitHub

git clone https://github.com/NCAR/gmtb-scm
cd gmtb-scm
git submodule init
git submodule update

At this point, you have checked out the correct branches of all three repositories. Each repository is connected to a remote destination (simply called remote in GitHub language), which points to the NCAR GitHub pages and is labeled as origin. For consistency with the CCPP workflow, we recommend renaming the NCAR remote destination to upstream for all repositories. For example:

cd ccpp/framework
git remote rename origin upstream
cd ../..

Checking out remote branches means that your local branches are in a detached state, since you cannot commit directly to a remote branch. As long as you are not making any modifications in a certain repository, this is not a problem. If during your development work changes are made to the corresponding upstream branch, you can simply navigate to this repository and check out the updated version (example ccpp-physics):

cd ccpp/physics
git remote update
git checkout upstream/dtc/develop
cd ../..

However, if you are making changes in a repository (submodule or main repository), you must create a local branch, for example in gmtb-scm:

git checkout -b my_local_development_branch

Once you are ready to contribute the code to the upstream repository, you need to create a PR (see next section). In order to do so, you first need to create your own fork of this repository (see previous section) and configure your fork as an additional remote destination, which we typically label as origin. For the same example gmtb-scm:

git remote add origin https://github.com/YOUR_GITHUB_USER/gmtb-scm
git remote update

Then, push your local branch to your fork:

git push origin my_local_development_branch

For each repository/submodule, you can check the configured remote destinations and all existing branches (remote and local):

git remote -v show
git remote update
git branch -a

As opposed to branches without modifications described in step 3, changes to the upstream repository can be brought into the local branch by pulling them down. For the example of submodule ccpp-physics (where a local branch is checked out):

cd ccpp/physics
git remote update
git pull upstream dtc/develop

7.3. Committing Changes to your Fork¶

Once you have your fork set up to begin code modifications, you should check that the cloned repositories upstream and origin are set correctly:

git remote -v

This should point to your fork as origin and the repository you cloned as upstream:

origin             https://github.com/YOUR_GITHUB_USER/ccpp-physics (fetch)
origin             https://github.com/YOUR_GITHUB_USER/ccpp-physics (push)
upstream   https://github.com/NCAR/ccpp-physics (fetch)
upstream   https://github.com/NCAR/ccpp-physics (push)

Also check what branch you are working on:

git branch

This command will show what branch you have checked out on your fork:

* features/my_local_development_branch
  dtc/develop
  master

After making modifications and testing, you can commit the changes to your fork. First check what files have been modified:

git status

This git command will provide some guidance on what files need to be added and what files are “untracked”. To add new files or stage modified files to be committed:

git add filename1 filename2

At this point it is helpful to have a description of your changes to these files documented somewhere, since when you commit the changes, you will be prompted for this information. To commit these changes to your local repository and push them to the development branch on your fork:

git commit
git push origin features/my_local_development_branch

When this is done, you can check the status again:

git status

This should show that your working copy is up to date with what is in the repository:

On branch features/my_local_development_branch
Your branch is up to date with 'origin/features/my_local_development_branch'.
nothing to commit, working tree clean

At this point you can continue development or create a PR as discussed in the next section.

7.4. Contributing Code, Code Review Process¶

Once your development is mature, and the testing has been completed (see next section), you are ready to create a PR using GitHub to propose your changes for review.

7.4.1. Creating a PR¶

Go to the github.com web interface, and navigate to your repository fork and branch. In most cases, this will be in the ccpp-physics repository, hence the following example:

Navigate to: https://github.com/<yourusername>/ccpp-physics

Use the drop-down menu on the left-side to select a branch to view your development branch

Use the button just right of the branch menu, to start a “New Pull Request”

Fill in a short title (one line)

Fill in a detailed description, including reporting on any testing you did

Click on “Create pull request”

If your development also requires changes in other repositories, you must open PRs in those repositories as well. In the PR message for each repository, please note the associate PRs submitted to other repositories.

Several people (aka CODEOWNERS) are automatically added to the list of reviewers on the right hand side. If others should be reviewing the code, click on the “reviewers” item on the right hand side and enter their GitHub usernames

Once the PR has been approved, the change is merged to master by one of the code owners. If there are pending conflicts, this means that the code is not up to date with the trunk. To resolve those, pull the target branch from upstream as described above, solve the conflicts and push the changes to the branch on your fork (this also updates the PR).

Note. GitHub offers a draft pull request feature that allows users to push their code to GitHub and create a draft PR. Draft PRs cannot be merged and do not automatically initiate notifications to the CODEOWNERS, but allow users to prepare the PR and flag it as “ready for review” once they feel comfortable with it.