.. _CodeManagement: ************************************************** CCPP Code Management ************************************************** ================================ Organization of the Code ================================ This chapter describes the organization of the code, provides instruction on the GitHub workflow and the code review process, and outlines the release procedure. It is assumed that the reader is familiar with using basic GitHub features. A GitHub account is necessary if a user would like to make and contribute code changes. ----------------------- UFS Atmosphere ----------------------- The :term:`UFS` Atmosphere source code is contained in the ufs-weather-model code repository. CCPP users and developers should use the ufs-weather-model code and the associated submodules maintained in GitHub. The umbrella repository is located at: https://github.com/ufs-community/ufs-weather-model There are nine submodules referenced in the ufs-weather-model repository: https://github.com/NCAR/ccpp-framework https://github.com/NCAR/ccpp-physics https://github.com/NOAA-EMC/fv3atm https://github.com/NOAA-EMC/GFDL_atmos_cubed_sphere https://github.com/NOAA-EMC/NEMS https://github.com/NOAA-EMC/FMS https://github.com/noaa-psd/stochastic_physics https://github.com/NOAA-EMC/WW3 https://github.com/NOAA-EMC/NCEPLIBS-pyprodutil Users have read-only access to these repositories and as such cannot accidentally destroy any important (shared) branches of these authoritative repositories. All of these repositories are public (no GitHub account required) and may be used directly to read or create forks. Write permission is generally restricted, however. The following table shows the default branch of each authoritative repository and are recommended for CCPP users and developers: +-----------------------------------------------------+-------------+--------------------------------------------+ | Repository (ufs-weather-model release version) | Branch name | Directory location under ufs-weather-model | +=====================================================+=============+============================================+ | https://github.com/ufs-community/ufs-weather-model | develop | `.` | +-----------------------------------------------------+-------------+--------------------------------------------+ | https://github.com/NCAR/ccpp-physics | master | FV3/ccpp/physics | +-----------------------------------------------------+-------------+--------------------------------------------+ | https://github.com/NCAR/ccpp-framework | master | FV3/ccpp/framwwork | +-----------------------------------------------------+-------------+--------------------------------------------+ | https://github.com/NOAA-EMC/fv3atm | develop | FV3 | +-----------------------------------------------------+-------------+--------------------------------------------+ | https://github.com/NOAA-EMC/GFDL_atmos_cubed_sphere | dev/emc | FV3/atmos_cubed_sphere | +-----------------------------------------------------+-------------+--------------------------------------------+ | https://github.com/NCAR/NEMS | develop | NEMS | +-----------------------------------------------------+-------------+--------------------------------------------+ | https://github.com/NCAR/FMS | master | FMS | +-----------------------------------------------------+-------------+--------------------------------------------+ | https://github.com/noaa-psd/stochastic_physics | master | stochastic_physics | +-----------------------------------------------------+-------------+--------------------------------------------+ | https://github.com/NOAA-EMC/WW3 | master | WW3 | +-----------------------------------------------------+-------------+--------------------------------------------+ | https://github.com/NOAA-EMC/NCEPLIBS-pyprodutil | develop | NEMS/tests/produtil/NCEPLIBS-pyprodutil | +-----------------------------------------------------+-------------+--------------------------------------------+ ----------------------- SCM ----------------------- CCPP developers should use the SCM code and submodules maintained by GMTB in GitHub. The ccpp-framework and ccpp-physics repositories are used by the two models, creating a two-way dependency between them (see :ref:`committing-changes` for implications on the code review and testing procedure). https://github.com/NCAR/gmtb-scm As with ufs-weather-model, there are two submodules referenced in the gmtb-scm repository: https://github.com/NCAR/ccpp-framework https://github.com/NCAR/ccpp-physics Users have read-only access to these repositories and as such cannot accidentally destroy any important (shared) branches of these authoritative repositories. Both CCPP repositories and the SCM repositories are public (no GitHub account required) and may be used directly to read or create forks. Write permission is generally restricted, however. The following branches are recommended for CCPP users and developers: +----------------------------------------+-------------+ | Repository (GMTB development version) | Branch name | +========================================+=============+ | https://github.com/NCAR/gmtb-scm | master | +----------------------------------------+-------------+ | https://github.com/NCAR/ccpp-physics | master | +----------------------------------------+-------------+ | https://github.com/NCAR/ccpp-framework | master | +----------------------------------------+-------------+ -------------------------------------- Directory Structure of ccpp/framework -------------------------------------- The following is the directory structure for the ccpp/framework (condensed version): .. code-block:: console ├── cmake # cmake files for building ├── doc # Documentation for design/implementation and developers guide │ ├── DevelopersGuide │ │ └── images │ └── img ├── schemes # Example ccpp_prebuild_config.py │ ├── check ├── scripts # Scripts for ccpp_prebuild.py, metadata parser, etc. │ ├── fortran_tools │ └── parse_tools ├── src # CCPP framework source code │ └── tests # SDFs and code for testing ├── test │ └── nemsfv3gfs # NEMSfv3gfs regression test scripts └── tests # Development for framework upgrades -------------------------------------- Directory Structure of ccpp/physics -------------------------------------- The following is the directory structure for the ccpp/physics (condensed version): .. code-block:: console ├── physics # CCPP physics source code and metadata files │ ├── docs # Scientific documentation (doxygen) │ │ ├── img # Figures for doxygen │ │ └── pdftxt # Text files for documentation ===================================================== GitHub Workflow (setting up development repositories) ===================================================== The CCPP development practices make use of the GitHub forking workflow. For users not familiar with this concept, this website provides some background information and a tutorial. --------------- Creating Forks --------------- The GitHub forking workflow relies on forks (personal copies) of the shared repositories on GitHub. These forks need to be created only once, and only for directories that users will contribute changes to. The following steps describe how to create a fork for the example of the ccpp-physics submodule/repository: Go to https://github.com/NCAR/ccpp-physics and make sure you are signed in as your GitHub user. Select the "fork" button in the upper right corner. * If you have already created a fork, this will take you to your fork. * If you have not yet created a fork, this will create one for you. Note that the repo name in the upper left (blue) will be either "NCAR" or "your GitHub name” which tells you which fork you are looking at. Note that personal forks are not required until a user wishes to make code contributions. The procedure for how to check out the code laid out below can be followed without having created any forks beforehand. ------------------------------------------ Checking out the Code (ufs-weather-model) ------------------------------------------ Start with checking out the main repository from the ufs-community GitHub: .. code-block:: console git clone https://github.com/ufs-community/ufs-weather-model cd ufs-weather-model git checkout ufs_public_release git submodule update --init --recursive At this point, you have checked out the correct branches of all tem repositories. Each repository is connected to a remote destination (simply called remote in GitHub language), which points to the ufs-community GitHub pages and is labeled as origin. For consistency with the CCPP workflow, we recommend renaming the ufs-community remote destination to upstream for any repositories that you will be modifying in order to guarantee that your development will be pushed to your fork and not directly to the ufs-community repository. For example: .. code-block:: console cd FV3/ccpp/framework git remote rename origin upstream cd ../../.. Checking out remote branches as submodules means that your local branches are in a detached head state, since the submodule points to a specific commit. As long as you are not making any modifications in a certain repository, this is not a problem. If during your development changes are made to the corresponding upstream branch, you can simply navigate to this repository and check out the updated version (example NEMS): .. code-block:: console cd NEMS git remote update git remote rename origin upstream git checkout upstream/ufs_public_release cd .. However, if you are making changes in a repository (submodule or main repository), you must create a local branch, for example in ufs-weather-model: .. code-block:: console git checkout -b my_local_development_branch Once you are ready to contribute the code to the upstream repository, you need to create a pull request (PR; see next section). In order to do so, you will use your own fork of this repository (see previous section) and configure your fork as an additional remote destination, which we typically label as origin. For the same example ufs-weather-model: .. code-block:: console git remote add origin https://github.com/YOUR_GITHUB_USER/ufs-weather-model git remote update Then, push your local branch to your fork: .. code-block:: console git push origin my_local_development_branch For each repository/submodule, you can check the configured remote destinations and all existing branches (remote and local): .. code-block:: console git remote -v show git remote update git branch -a As opposed to branches without modifications described in step 3, changes to the upstream repository can be brought into the local branch by pulling them down. For the example of submodule FV3 (where a local branch is checked out): .. code-block:: console cd FV3 git remote update git pull upstream ufs_public_release ----------------------------------- Checking out the Code (SCM) ----------------------------------- The process for checking out SCM is described in the following, assuming access via https rather than ssh. We strongly recommend setting up passwordless access to GitHub (see https://help.github.com/categories/authenticating-to-github). Start with checking out the main repository from the NCAR GitHub .. code-block:: console git clone https://github.com/NCAR/gmtb-scm cd gmtb-scm git submodule init git submodule update At this point, you have checked out the correct branches of all three repositories. Each repository is connected to a remote destination (simply called remote in GitHub language), which points to the NCAR GitHub pages and is labeled as origin. For consistency with the CCPP workflow, we recommend renaming the NCAR remote destination to upstream for all repositories. For example: .. code-block:: console cd ccpp/framework git remote rename origin upstream cd ../.. Checking out remote branches means that your local branches are in a detached state, since you cannot commit directly to a remote branch. As long as you are not making any modifications in a certain repository, this is not a problem. If during your development work changes are made to the corresponding upstream branch, you can simply navigate to this repository and check out the updated version (example ccpp-physics): .. code-block:: console cd ccpp/physics git remote update git checkout upstream/dtc/develop cd ../.. However, if you are making changes in a repository (submodule or main repository), you must create a local branch, for example in gmtb-scm: .. code-block:: console git checkout -b my_local_development_branch Once you are ready to contribute the code to the upstream repository, you need to create a PR (see next section). In order to do so, you first need to create your own fork of this repository (see previous section) and configure your fork as an additional remote destination, which we typically label as origin. For the same example gmtb-scm: .. code-block:: console git remote add origin https://github.com/YOUR_GITHUB_USER/gmtb-scm git remote update Then, push your local branch to your fork: .. code-block:: console git push origin my_local_development_branch For each repository/submodule, you can check the configured remote destinations and all existing branches (remote and local): .. code-block:: console git remote -v show git remote update git branch -a As opposed to branches without modifications described in step 3, changes to the upstream repository can be brought into the local branch by pulling them down. For the example of submodule ccpp-physics (where a local branch is checked out): .. code-block:: console cd ccpp/physics git remote update git pull upstream dtc/develop .. _committing-changes: ================================== Committing Changes to your Fork ================================== Once you have your fork set up to begin code modifications, you should check that the cloned repositories upstream and origin are set correctly: .. code-block:: console git remote -v This should point to your fork as origin and the repository you cloned as upstream: .. code-block:: console origin https://github.com/YOUR_GITHUB_USER/ccpp-physics (fetch) origin https://github.com/YOUR_GITHUB_USER/ccpp-physics (push) upstream https://github.com/NCAR/ccpp-physics (fetch) upstream https://github.com/NCAR/ccpp-physics (push) Also check what branch you are working on: .. code-block:: console git branch This command will show what branch you have checked out on your fork: .. code-block:: console * features/my_local_development_branch dtc/develop master After making modifications and testing, you can commit the changes to your fork. First check what files have been modified: .. code-block:: console git status This git command will provide some guidance on what files need to be added and what files are “untracked”. To add new files or stage modified files to be committed: .. code-block:: console git add filename1 filename2 At this point it is helpful to have a description of your changes to these files documented somewhere, since when you commit the changes, you will be prompted for this information. To commit these changes to your local repository and push them to the development branch on your fork: .. code-block:: console git commit git push origin features/my_local_development_branch When this is done, you can check the status again: .. code-block:: console git status This should show that your working copy is up to date with what is in the repository: .. code-block:: console On branch features/my_local_development_branch Your branch is up to date with 'origin/features/my_local_development_branch'. nothing to commit, working tree clean At this point you can continue development or create a PR as discussed in the next section. ========================================= Contributing Code, Code Review Process ========================================= Once your development is mature, and the testing has been completed (see next section), you are ready to create a PR using GitHub to propose your changes for review. ------------------- Creating a PR ------------------- Go to the github.com web interface, and navigate to your repository fork and branch. In most cases, this will be in the ccpp-physics repository, hence the following example: | Navigate to: https://github.com//ccpp-physics | Use the drop-down menu on the left-side to select a branch to view your development branch | Use the button just right of the branch menu, to start a “New Pull Request” | Fill in a short title (one line) | Fill in a detailed description, including reporting on any testing you did | Click on “Create pull request” If your development also requires changes in other repositories, you must open PRs in those repositories as well. In the PR message for each repository, please note the associate PRs submitted to other repositories. Several people (aka CODEOWNERS) are automatically added to the list of reviewers on the right hand side. If others should be reviewing the code, click on the “reviewers” item on the right hand side and enter their GitHub usernames Once the PR has been approved, the change is merged to master by one of the code owners. If there are pending conflicts, this means that the code is not up to date with the trunk. To resolve those, pull the target branch from upstream as described above, solve the conflicts and push the changes to the branch on your fork (this also updates the PR). Note. GitHub offers a draft pull request feature that allows users to push their code to GitHub and create a draft PR. Draft PRs cannot be merged and do not automatically initiate notifications to the CODEOWNERS, but allow users to prepare the PR and flag it as “ready for review” once they feel comfortable with it.