流线型：并行软件开发的分支模式（Streamed Lines: Branching Patterns for Parallel Software Development）
摘要：绝大部分软件版本控制系统提供了多重开发分支、将源代码从一个开发分支合并到另一个分支的机制。但是，人们通常会误用或者不完全理解使用这些机制的技术、方针和指示。这是不幸的，因为错误地使用分支和合并会中断并行软件开发项目。“流线”是一种模式语言，它能将相关的开发生产线合理地组织成代码变更的“分叉-汇合”流。
| Brad Appleton | <bradapp@enteract.com> | Motorola Network Solutions Group |
| Stephen P. Berczuk | <berczuk@acm.org> | NetSuite Development |
| Ralph Cabrera | <cabrerar@agcs.com> | AG Communication Systems |
| Robert Orenstein | <rlo@perforce.com> | Perforce Software |
Copyright © 1998 by Brad Appleton, Stephen Berczuk, Ralph Cabrera, and Robert Orenstein. Permission is granted to copy for the PLoP '98 conference.
Abstract: Most software version control systems provide mechanisms for branching into multiple lines of development and merging source code from one development line into another. However, the techniques, policies and guidelines for using these mechanisms are often misapplied or not fully understood. This is unfortunate, since the use or misuse of branching and merging can make or break a parallel software development project. Streamed Lines is a pattern language for organizing related lines of development into appropriately diverging and converging streams of source code changes.
Keywords: Branching, Parallel Development, Patterns, Software Configuration Management, Version Control
| Send us your comments! | |
Read this section by following the above hyperlink if you want an introduction to SCM patterns. You can read about our motivation and progress in developing an SCM pattern language, and view a diagram showing the relationships between SCM patterns. Skip ahead to the next section if you want to stay focused on parallel development and branching.
Any software project of certain team and system sizes will invariably require at least some efforts to be conducted in parallel. Large projects require many roles to be filled: developers, architects, build managers, quality assurance personnel, and other participants all make contributions. Multiple releases must be maintained, and many platforms may be supported. It is often claimed that parallel development will boost team productivity and coordination, but these are not the only reasons for developing in parallel. As [Perry98] points out, parallel development is inevitable in projects with more than one developer. The question is not "should we conduct a parallel development effort", but "how should a parallel development effort best be conducted?"
[Perry98] suggests that many of the basic parallel development problems which arise can be traced back to the essential problems of: system evolution, scale, multiple dimensionality, and knowledge distribution.
Evolution compounds the problem of parallel development because we not only have parallel development within each release, but among releases as well.
Scale compounds the problem by increasing the degree of parallel development and hence increasing both the interactions and interdependencies among developers.
Multiple dimensions of system organization compounds the problems by preventing tidy separations of development into independent work units.
Distribution of knowledge compounds the problem by decreasing the degree of awareness in that dimension of knowledge that is distributed.
Thus, a fundamental and important problem in building and evolving complex large scale software systems is how to manage the phenomena of parallel changes. How do we support the people doing these parallel changes by organizational structures, by project management, by process, and by technology?
If parallel development is a fact of life for any large software project, then how can developers making changes to the system in parallel be supported by project management, organizational structures, and technology? Streamed Lines is a pattern language that attempts to provide at least a partial answer to this question by presenting branching and merging patterns for decomposing a project's workflow into separate lines of development, and then later recomposing these lines back into the main workstream. The patterns describe recurring solutions for deciding how and when development paths should diverge (branch) and converge (merge).
Streamed Lines does not describe a complete solution to all the problems encountered during parallel development; It merely attempts to reveal the ways in which branches can be used to help create an effective parallel development solution. What do we even mean by "effective parallel development"? [Atria95] defines effective parallel development as:
... the ability for a software team to undertake multiple, related development activities -- designing, coding, building, merging, releasing, porting, testing, bug-fixing, documenting, etc. -- at the same time, often for multiple releases that use a common software base, with accuracy and control.
Note that this definition extends to include teams that span multiple locations, an increasingly common situation for many organizations. It encompasses all elements of a software system and all phases of the development lifecycle. Inherent within the definition is the concept of integration, in which parallel development activities and projects merge back into the common software base. Also, the definition of effective parallel development includes process control -- the policies and "rules of the road" that help assure a controlled, accurate development environment.
So how can branching help us achieve effective parallel development? Branches may be used to isolate changes, and to insulate developers from other's integrated changes that have yet to be integrated, built, tested, and baselined. Branches may also be used to organize the decomposition work into change-tasks and work-streams and to control the integration of changes from tasks and streams into other streams. When used appropriately in this manner, branching helps address problems of communication, visibility, project planning and tracking, and ultimately risk management.
But branching is most conceptually powerful when viewed from a project-wide or system-wide perspective; the resultant version tree reflects the evolution of an entire project or system. We call this project-oriented branching. With project-oriented branching, branches are used and organized and viewed in the context of an entire project, product, or system. Project-oriented branching imposes a more or less uniform structure on the version trees for all the files in the system. Instead of emphasizing modifications to individual files, project-oriented branching focuses primarily on the flow of logical changes across the entire system. Logical changes flow through and between streams of work in which product and component versions are integrated, built, baselined, and released.
It should also be mentioned that using branches for more than 2-3 of these dimensions at the same time is discouraged because it can necessitate a combinatorial explosion of branches spawned from the same origination point (which is quite unwieldy). [Conradi96] discusses this inherent weakness of hierarchical branching and version-trees: a hierarchical organization is often convenient, but it quickly breaks down when variance occurs simultaneously along multiple dimensions.
When drawing codelines, branches, change-tasks, and their relationships, we use a tree structure with branch-names inside boxes and version-names inside circles (a "box" or "circle" with no name inside is considered "anonymous"). Branches and codelines are indicated with solid lines, whereas merges and propagations are indicated with dashed lines. These version-tree diagrams are reminiscent of interaction sequence diagrams in the UML; but we draw the timeline from left to right instead of from top to bottom (to conserve space).
Branch names always appear at the beginning of the timeline for the branch, and are preceded by a '/'. A "box" appearing in the middle of a timeline for a branch corresponds to a change-task that was performed "on-line" (directly on the codeline, instead of on its own branch), and there is no leading slash in front of the name for such a change-task. The length of a change-task "box" may be used to indicate its duration relative to other change-tasks.
|
| ||||
|
| ||||
| |||||
The participants in Streamed Lines are distributed among these four categories as follows:
| Basic Branch/Line Elements | Branching Policy Patterns |
|---|---|
|
|
| Branch Creation Patterns | Branch Structuring Patterns |
|
The full pattern descriptions appear in Appendix A.
Generally speaking, using more branches for greater isolation reduces safety risks, but at the expense of more merging and integration effort. More merging and integration also requires more communication and greater visibility of changes and baselines. Using fewer branches reduces merging and integration efforts, but at the expense of less isolation and less safety. Merging sooner rather than later fleshes out risks early on while there is more time to address them, but requires continual efforts to regularly monitor and address such risks.
In short, you will have to confront and manage risks concerning safety, productivity, and communication no matter what you do. Time and effort must be invested to manage these risks. The three basic ways to do this are to pay now, to pay later, or to pay-as-you-go.
The most productive overall strategies attempt to invest a reasonably small amount up front, and then pay the rest as they go. The larger and more critical and risk-averse your project is, the more you will need to invest in "up front" planning and policies, while still employing a pay-as-you-go strategy throughout the lifetime of the project (which includes regular monitoring and feedback to make incremental corrections). Such an approach essentially tries to offload back-end costs (of deferred or unmanaged risks) by handling the most critical risks "up front" as a minimal initial investment, and to amortize the remaining costs using a "just-in-time" approach.
Here then are the important strategic decisions to make while planning the branching and merging road-map for your parallel development efforts. Be aware that performing less up-front planning requires more attentive and visible monitoring and feedback; while more up-front planning often results in more things that need to be corrected later on. These differences should decrease, and eventually converge, as the project evolves and its parallel development policies and procedures become more stable and mature.
Typically, the most fundamentally important tradeoff to consider will be that of safety versus liveness. To get an idea of how much safety risk you can tolerate, ask yourself how much time and effort is required to back-out an unwanted or detrimental change from one of your codelines and builds. How many people does it impact and how soon (and how critically) are they impacted? How much rework and rebuilding is required and how much time and staff are required to perform that rework? How much additional communication overhead does the rework impose?
If the answer to these questions leads you to believe it would be a very significant, or even monumental undertaking to back-out an unwanted change, then your project probably has a very low threshold for safety risks. If on the other hand it seems that only a select few people would be affected and it wouldn't take very much time to correct the problem, then you may have a very high threshold for safety risks.
Don't forget to consider how your risk-threshold will change and evolve as the project evolves and matures! It is exceedingly common for a project to tolerate more risk (and sometimes have greater time-to-market pressures) before it has been deployed to a broad base of customers than after it has been deployed and several releases are being supported and maintained. Also, if the size of the team or of the system is expected to grow considerably, it may make more sense to take some preventive measures early on, before it becomes to difficult to impose non-trivial changes in the team's process and behavior. At the very least, you will need to plan to migrate from a process that tolerates more risk to a process that eventually tolerates less risk.
The choice of early or deferred branching also affects the visibility with which teamwork and workflow can be communicated from a file's version tree. Deferred branching may hide the intent of a change or set of changes to go into specific releases. Early branching makes this intent clear early on, but requires more effort to follow through with that intent and propagate the change to more codelines than would be required if you had waited longer before branching.
The branching style that you decide is best suited for your environment will dictate a complementary set of patterns and pattern variants:
| Early Branching Style | Deferred Branching Style |
|---|---|
|
|
Regardless of the branching style selected, Codeline Policy and Codeline Ownership should used be for every branch and codeline created. These two practices need to be employed in a way that is readily visible to the team, and which can be easily and quickly communicated in as short a time-span as possible.
Patterns like Parallel Maintenance/Development and Overlapping Releases are typically the first branching structures many shops encounter. They can be applied using either branching-style. It depends primarily upon when you branch (early or late) and upon which effort goes on the branch and which stays on the parent codeline.
Early branching tends to keep the release or major release as the invariant for each codeline. So instead of splitting development and maintenance across codelines, it keeps the same release on the same codeline, regardless of whether or not it is development effort or maintenance effort for the given release.
For deferred branching, the releasing/maintenance effort will always be the one that branches off, allowing the latest and greatest development to continue on the same line as before. This way of thinking may be peculiar to those accustomed to an early branching style that uses separate codelines for each release; they may have difficulty understanding why it is coherent. With deferred branching, it's not the release that remains invariant on the branch, it's that the recency of the effort on the branch: the latest development efforts, or else the latest maintenance efforts.
Although the choice of merging style often follows from the chosen branching style, a higher risk branching style does not necessarily imply a higher risk merging style. In fact, you may wish to offset high risk in one with low risk in the other. If you take more risks when splitting things apart, you may want to take less risk when putting things back together.
Remember that every time you add another line of integration, you are in effect, adding another level of indirection: you gain more isolation and nicer conceptual organization but you spend more time merging. It should be noted that a Virtual Codeline is somewhat merge-evasive and may be used to simulate just about any kind of codeline. The merging patterns that are more suited to each merging style are as follows:
| Relaxed Merging Style | Restricted Merging Style |
|---|---|
|
|
In either case, frequent incremental integration is always a good idea (using Merge Early and Often or one of its variants) but the merging frequency and ownerships will differ between the two styles. The relaxed style favors liveness and assumes higher risk by having people merge and propagate their own changes across codelines. The more restricted style favors safety and has more codelines, each with more restricted access, and with codeline-owners performing most of the merges.
Unlike the branching styles, the merging styles may be mixed and matched to achieve a gradual progression from high-activity codelines with relaxed policies to lower-activity codelines with restricted policies. This can be accomplished with patterns such as Docking Line, Subproject Line, Component Line and Remote Line. But with a more relaxed style, each of these kinds of codelines will typically merge back to the development line while a more restricted style is more likely to use it as one in a set of Staged Integration Lines.
Once again, one or more of the following merging patterns will be used with the above: MYOC, Docking Line, or Staged Integration Lines.
You may need them very rarely, or only for certain kinds of projects and project teams. But when the project does require them, they often have a very profound impact on the overall shape of the project-wide version tree, and on the overall organization of parallel development efforts. These patterns (along with Change Propagation Queues) should be used sparingly, and only as the need arises. This is especially true of platform-lines since it is often better to handle multi-platform issues with separate files and/or directories than with separate branches.
Not only does early and frequent integration flesh out risk sooner and in smaller "chunks," it also communicates changes between teammates. Every time a developer integrates a new baseline into their workspace, or a new change into the baseline, they learn something about what has happened to the system and where it has changed. In this sense, integration turns out to be a very real form of communication, albeit an indirect one. For this reason, it is crucial that the presence of new baselines and baselevels are clearly and visibly communicated to all concerned, and that the completion of important changes that are ready to be built/baselines are also clearly and visibly communicated.
So perhaps a corollary to "integrate early and often" would be "commit changes visibly and clearly." This includes changes that have been committed to be included into a particular baseline/codeline, as well as baselines that are now ready to be sync-ed into developer's workspaces.
This will help reduce risk by isolating variation along the appropriate dimension of work. While this does help to control and contain the amount of variation to a locally manageable region, it does impose an additional integration burden later on. (So does branching on incompatibility.) The theory here is that the integration overhead at the end will be minimized by the continual control that is more easily afforded by isolating the change.
But don't use branches to solve all your problems! Many problems are best addressed by different means. For example, numerous multi-platform issues are better solved by using extra files and directories rather than platform-branches. Don't use branches as a "hammer" to make every problem look like a nail, and don't "sow" a new branch unless you can reap the benefits.
Preserve the physical integrity of the branch! Don't merge incomplete or inconsistent changes into the codeline; and don't leave codelines in inconsistent states. When the configuration of a codeline is inconsistent or incorrect it can adversely impact all users of the codeline. Try to keep codelines reliably consistent, and consistently reliable.
Choose optimistic or pessimistic branching policies and stick with them! For a given project, strike a sensible balance of trade-offs between safety (isolation, access control, code integrity, and risk mitigation) and liveness (productivity, integration overhead, working "on-line") and then apply them in a consistent manner. The balance may need to be dynamically adjusted over time; but at any given time, the policies should be consistent with one another.
If you isolate people from their work, systemic disconnection may result: developers lose touch with the effects of their own efforts on the overall project. If you segregate people from each other according to their work tasks, social isolation may occur: people lose touch with one another and with the overall project team. The purpose of parallelization is not to isolate people from people, or people from their work, but to isolate work from other work. Conway's Law (see [Cope95]) applies just as much to the architecture of the project's version tree as it does to the architecture of the system. Use this wisdom to your advantage (and ignore it at your peril).
|
|
As [Lea96] describes, some of the most basic tradeoffs to be made when designing concurrent object systems are those of safety ("The property that nothing bad ever happens") and liveness ("The property that anything ever happens at all"). These tradeoffs are essentially the same for software development:
From either direction, the goal is to assure liveness across the broadest possible set of contexts without sacrificing safety.
The need to apply such strategies across the broadest possible set of contexts ties into their reusability across the project, and between projects. Hence all the same issues and concerns mentioned by [Lea96] regarding safety, liveness, and reusability also arise during parallel development.
In effect, every codeline and branch represents a form of risk management by isolating how functionality, environment, knowledge, teamwork, responsibility, and reliability, are distributed and disseminated across time and space.
Branching also helps communication and collaboration be effectively organized, synchronized, and parallelized. If used properly so that it isolates work instead of people, branching promotes effective teamwork and really can reduce time-to-release. If you thoughtfully apply risk-aware strategies for the selection of branching and merging styles, and periodically take a step back to review and revise the overall branching-tree, you should be able to reap the benefits of parallel development (shorter cycle-time) and keep the amount of synchronization overhead (and risk) to a manageable level.
Using file-oriented branching to represent project-oriented branching results in a fair amount of trivial merging where revision contents need to be propagated from branch to branch with little or no difference between them (often causing unnecessary rebuilds when in fact file-contents have not changed between revisions). Good merging tools can minimize the pain and overhead associated with this, but the overhead can still be significant.
Unfortunately, the majority of readily available VC tools don't provide the user with anything better. It would be far more suitable if one's VC or SCM tool provided predefined constructs which directly map to the conceptual notions of: change-sets, activities, and activity-streams, without being dependent upon branches. Then we could use the SCM tool to directly model parallel effort and workflow and let the tool itself worry about how to handle the low-level concurrency control (branching) with the help of some user-supplied policy preferences. There are a select few tools which actually do provide this capability but they are presently in the minority. So unless you are using such a tool, branching tends to be the next best mechanism for supporting parallelism.
What this means is that tool-generated diagrams and queries/reports can show version trees which closely conform to the intended work breakdown structure (WBS) for the project team. This helps visibly track and communicate status and progress in "real-time" to all users of the VC tool and repository.
The branching tree of a project represents the structure of its evolution in terms of change-flows. The flow of work activities is also an important project structure. Streamed Lines attempts to coordinate these two sets of structures so that activity and workflow conveniently map to change-flows (using branches as the grouping mechanism). This helps makes the project's development and evolution easier to conceptualize and manage. In this manner, Streamed Lines assists in bringing some of the architectural and management structures of a software project into alignment.
[back to the table of contents]