Section VII. ROBOTIC SYSTEMS IN PRACTICE

Chapter 27. Systems Engineering

Once you graduate from university and start in the robotics workforce, you will be exposed to a massively different world than you've encountered in your classes, educational competitions like FIRST robotics, team projects in student organizations, and even research projects at well-reknowned labs. You may be thinking, "Well, I've worked on team class projects, so I know pretty much how this will go. Some people pull their weight, some slack off, but, in the end, everything will go ok for our final report. The workforce will be pretty much like that." (Oh, how I wish to be so young and naive again!)

Don't underestimate it: the scale of the engineering effort (and its impact) in the enterprise setting will be larger than anything else you have experienced at university, the impact of the "soft skills" like teamwork and communication will be much higher, and the quality standards for your technical contributions will be higher too. It is not uncommon to feel some culture shock during this transition. Hopefully you will have had some summer internships doing real R&D to help you prepare for this experience. Or, you may be lucky enough to participate in one of the few university labs that engages in system engineering at a reasonable scale -- and by "reasonable" I mean 10+ simultaneous developers on one project. Even if you're a student with a 4.0 GPA, if you can't adapt to the complexities of systems engineering, you might end up a perpetual junior engineer bumbling your way around an organization with no hope for career advancement.

Real-world robotics engineering requires working on large and diverse teams over long periods of time. A good engineer is grounded in 1) the theory governing component algorithms, 2) system integration and development practices, and 3) effective communication skills to document, justify, and advocate for their work. Teams of engineers also need managers, and a good manager is grounded in 4) logical and organized thought about the system at appropriate abstraction levels, 5) project management skills to plan, assign, and track development progress, and 6) people skills to motivate the engineering team and convey the vision and progress to upper-level management. Both classes of employees should also bring a sense of personal investment in the project so that they stay enthusiastic as setbacks are encountered, the project scope changes, and personnel changes occur. Although a lot of these aspects cannot be taught outside of self-help books, we will be able to provide some degree of training in items 2), 3), 4), and 5) in this book.

This chapter provides a brief overview of theory, processes, project management, and organizational practices of typical robotic systems engineering projects. We can only scratch the surface of this material, as there have been many wonderful books written about systems engineering, software engineering, organizational strategy, and organizational psychology. People can be quite opinionated and passionate about these topics, and we lack hard data exploring which methods are more successful than others, so it's best not to delve too deep into any one philosophy. Nevertheless, this high-level summary should be help the aspiring robotics engineer (and engineering manager) predict the terminology, best practices, and pain points they are expected to encounter in their future career.

Systems engineering theory

Abstraction

It is hard to define precisely what a "system" means, but for the most part we can settle on a somewhat vague meaning: a system is an artifact composed of multiple interacting components that is engineered for a defined purpose. Often (but not always) these components correspond to different physical units, computational devices, or pieces of code. The most critical aspect of this definition is that the components themselves are engineered to produce specified function by interacting with other components within the system. We can system as a network of components interacting through edges (a system diagram) and reason about operations and information flow at a more abstract level than thinking about the details of how each component is implemented. At an organizational level, we can also think about projects in terms of a timeline of implementing components, measuring their performance, or replacing old implementations with new ones.

---

Figure 1. A system diagram for a hypothetical perception system for an autonomous vehicle. The rounded boxes denote components of the system, and the boxes denote data that are components' inputs or outputs. The green boxes denote an interface to the vehicle hardware, the orange boxes are static elements or external to the behavior system, and the blue boxes denote behavior system code.

---

Abstraction is the key tool we use in system engineering to manage complexity. Abstraction is also hammered home in typical computer science curricula due to the complexity of large software projects. Its purpose is a cognitive one: human brains are simply incapable of reasoning holistically about thousands or millions of lines of computer code interacting with controllers, power electronics, motors, and mechanisms. Instead, for our own benefit we must break the system into smaller components, each of which fulfills a specific function. These "functions" are our mental model of how each component behaves or should behave. The mechanism by which we achieve abstraction is called encapsulation, which means hiding details of the implementation from the external user. We should not need to know all the details by which a motion planner works in order to use it, e.g., if it uses RRT, PRM, trajectory optimization, etc. We just need to know the inputs, the outputs, and its expected peformance.

Note that in a sufficiently complex system, the components are usually also systems themselves, built out of sub-components! You may ask, why do we choose one level of abstraction over another? One could define a car as a system of tens of thousands of parts down to the last bolt, but for most purposes that is not as useful of an abstraction as defining a car as a body, frame, engine, wheels, steering, electrical system, and passenger compartment. Useful for whom? Well, the company management, engineers, factory workers, parts suppliers, certification agencies, repair shops, and customers would tend to think of different parts of the vehicle that way. Indeed, the theory, expertise, design, tooling, and operation of each of these components is specialized for their specific function.

As a systems engineer, you may welcome abstraction at times, but at others, you may struggle against it. Some possible pitfalls include:

• Compatibility conflicts
• Incorrect abstractions
• Leaky abstractions
• Overzealous abstractions
• Bad abstractions

Considering again the car example, if you are designing a sleek and streamlined body with decorative elements that you know will sell to customers, you may run into a struggle with the engine designer who can no longer fit a sufficiently beefy engine to give the customers the horsepower they desire. This is a compatibility conflict which needs clever engineering or strong management to resolve. (If you are Ferrari, your boss tells you to quiet down and design the body around the engine!)

An incorrect abstraction is one in which one's mental model of the system may not be satisfied by the implementation. As a real-world example, my lab struggled with an issue for several days during development for the Amazon Picking Challenge. We found that when we were testing at certain times of the day, our robot would start acting strange and the picking performance would drop precipitously. Then we'd test again, and everything would work fine. The culprit? The Intel RealSense cameras we had at the time would normally report RGBD data at 30 frames per second (fps) in good lighting, but then silently drop to 15 fps in poor lighting. Because the students on the team would work long into the night, they set up the perception system to work appropriately with the lower frame rate. But at the higher frame rate, some network buffers were being filled with too RGBD images, and so the perception system was processing stale data from multiple seconds in the past. The issue here was that our working mental model of the camera was a device that provided data at a consistent rate, and this abstraction was not incorrect. Perhaps we should have read the documentation better or constructed more thorough unit tests!

Leaky abstractions are a similar concept that can cause all sorts of frustration. In the Amazon Picking Challenge, the variable frame rate of the camera caused side-effects that we did not account for, as we did not carefully design the perception system in mind with all the details of the ROS communication system. This is because the publish-subscribe abstraction used by ROS is, coarsely speaking, "a publisher sends a message and immediately the subscriber(s) get it". In order to find the issue the developer needs to know more about networking than was promised -- specifically ROS queues and the slow receiver problem. Once we found the culprit, the fix was easy (shortening the queues to only provide the latest data), but placing blame on the right component was tricky. (We'll see more about how to assign blame to components later.)

An overzealous abstraction occurs when a component is designed to encapsulate too much functionality. Developers of other components would like to interact with a finer level of control over its internal functions. For example, developers of industrial robots often provide a "go-to" subroutine that does not terminate until the robot arrives at its destination (or encounters a fault). This would not be acceptable if you wished to build a collision avoidance system that could stop the robot mid-motion if an obstacle were detected in the robot's path. A similar concept is the bad abstraction, in which a component tries to do a collection of things whose grouping is poorly rationalized or cognitively complex. Bad abstractions often come from a combination of overzealous encapsulation and changing requirements. As new use cases arise, the developer adds more and more configuration parameters to customize how the component functions, leading to an unwieldy, confusing set of inputs.

An aspect of abstraction that is somewhat unique to robotics is that many upstream components must model downstream components in order to function properly. For example, state estimators, object trackers, and planners need a dynamics model to predict how the system moves. If the movement mechanisms or low-level controllers grow in complexity, the dynamics become more complex, necessitating more complex models. Similarly, increased sensor capabilities usually lead to greater complexity in observation models used in state estimation or active sensing. For this reason, as we seek to improve component performance, we usually pay the price in terms of model complexity.

System diagrams and system specifications

The most important part of organizing a team of engineers is to build a shared mental model of what that function that system should perform, what components the system will consist of, and how those components will operate. There are many ways to build such mental models, listed in order of formality:

1. Background knowledge (a.k.a. book learning): all the topics studied in courses, books, and academic papers. After you have graduated from a robotics program, you should generally know the functions of forward and inverse kinematics, motion planning, trajectory optimization, Kalman filters, deep neural networks, etc.
2. Experiential knowledge: information that an individual gathers from interacting with the system and its components.
3. Community knowledge: information scraped from web forums. This includes the use of ChatGPT and other AI tools, which are trained on such community information. (Such information is often of dubious validity)
4. Tribal knowledge (a.k.a. institutional memory): information passed between team members and held within individuals' memory.
5. Textual documentation: design documents, code comments and documentation, technical manuals, presentations shared amongst the organization.
6. System diagrams: control flow diagrams, computation graphs, and state machines.
7. System specifications: application programming interfaces (APIs), interface definition languages (IDLs), behavior trees (BTs), and modeling languages, e.g., Universal Modeling Language (UML).

As a general rule, information should flow down this list toward documentation, diagrams, and specifications. As the formality of such information grows, it becomes more precise, interpretable, widely disseminated, and longer-lasting. The tradeoff is that turning information from mental information to formal knowledge takes time and effort. Keeping formal knowledge up-to-date is also more time-consuming.

Control flow diagrams

TODO

Computation graphs

TODO

---

Figure 3. A system diagram for a hypothetical planning system for an autonomous vehicle, with left-side inputs corresponding to the outputs of the perception system. It is clear from the diagram that the data processing occurs in a sequential (serial) manner, where higher-level concerns like deciding on the vehicle's mission and route are processed before lower-level ones like the vehicle's path or control outputs. What are the potential benefits and drawbacks of a serial system architecture?

---

State machines

TODO

Behavior trees

TODO

Reliability and redundancy

Given 𝑛 components in sequence, any one of which may fail independently with probability $\epsilon$, the probability that any one of them fails is $1−(1−\epsilon)^𝑛$

• Example: $\epsilon$=0.05, 𝑛=5 => 23% probability of failure
• Example: $\epsilon$=0.01, 𝑛=10 => 9% probability of failure

Given 𝑚 (redundant) components in parallel, the probability that all of them fails is $\epsilon^𝑚$

• Example: $\epsilon$=0.05, 𝑚=3 => 0.01% probability of failure
• Example: $\epsilon$=0.5, 𝑚=5=> 3% probability of failure

TODO figure

Now, it should be cautioned not to directly use these equations to predict true system failure probabilities, because component failures are often not independent. Suppose that in the spirit of redundancy, we have outfitted a drone with 3 inertial measurement units (IMUs) so that we have two backups in case any one fails. Each one may fail with a probability $<$ 1%, so we should expect our system to fail with probability $<$ 0.0001%, right? Well, if the IMUs rely on GPS readings for global positioning or on Earth's magnetic field for a compass heading, all three IMUs may be susceptible to GPS denial (indoor environments, tall buildings, or jamming), GPS blackouts, and magnetic interference. Or, if the drone gets jerked around rapidly, accelerometers may saturate leading to degradation of accuracy. So, a developer should watch out for common causes of simultaneous failure.

Nevertheless, these equations give a compelling rationale for three high-level goals in system development:

1. Improve the reliability of individual components(reduce $\epsilon$).
2. Minimize long chains of dependent components (minimize $n$).
3. Implement redundant approaches for the same task (maximize $m$).

You may ask, what strategies should a development team pursue to accomplish each of these goals? Addressing the first is fairly straightforward: find the "weakest links" in a sequence, and get your domain specialists to improve robustness ("harden") those components. Unit testing is an important practice to adopt here.

To address the second goal, we find that in robotics it is often impossible to reduce chains of dependencies past perception, planning, and control steps. There have been research efforts to perform "end-to-end" learning that circumvent intermediate steps, but these approahes have not yet reached the reliability and customizability of the classical sense-plan-act framework. On the other hand, we have seen major advances in perception where classical pipelines that have involved long chains of processing steps (e.g., from pixels to features, from features to parts, from parts to objects) have been replaced by deep neural networks. Also, complex planning pipelines that involve mission planning logic, task sequencing, subgoal definition, and low-level planning can be replaced with unified components, such as task-and-motion planning. It can also be helpful for an upstream planner to define a scoring function that rates terminal states rather than a single goal for a downstream planner. This is because an upstream planner can make a mistake in assigning an infeasible goal, and the downstream planner would be unable to find a solution. If, instead, the upstream planner assigns scores for possible goals (penalizing unfavorable goals) then the downstream planner has more options: it could find a less favorable but feasible solution.

Approaches for the third goal depend on whether the component's failures are a) reported by the component, b) due to random phenomena, such as sensor noise or a mechanical device wearing out, or c) systematic errors, such as sensor artifacts or an algorithm failing to find a valid solution. In cases a) and b) the replication approach simply adds duplicate units. If failures are reported (case a), then it is a simple matter of switching to a backup when a primary unit fails. If they are not detectable, then you will need a mechanism to estimate which units are failing, such as taking the median of 3 or more sensors or adding a separate anomaly detector. The median approach is an effective way of handling malfunctioning sensors which may report extreme values, since the median of 3 sensors will be one of the value from the remaining 2 functioning sensors. In case c), replication is insufficient since each unit will fail in the same way, i.e., each unit's errors would be affected by a common cause. Instead, you should implement alternative approaches that fail in different conditions than the primary approach. For example, for high-reliability scenarios like autonomous driving it is a good idea to consider implementing multiple redundant planners (e.g., generate-and-score, graph-search, sampling-based planners, and trajectory optimization) which are run simultaneously on different CPU cores. The resulting paths can be then rated and the best one chosen for execution.

The robot engineering process

A developer's responsibilities

• To create system features that are useful to users or other developers
• To characterize and report on the behavior of the system or its components
• To maintain desired functions of the system or its components under changing requirements
• To improve developer velocity, i.e., the rate at which developers contribute to items 1-3
• To enable and aid the usefulness of the system or its components through documentation and organization.

Phases of development

Generally speaking, a robotics project will follow the four phases listed here. If the organization is lucky, these steps and phases proceed one after another without a hitch. But, I have never heard of such a case in my life, and never do expect to hear of one! We will discuss caveats to this outline below.

Phase I: Planning

Product team	System integration team
Requirements gathering	System architecture design

Phase II: Component development

Hardware team	Perception team	Dynamics and control team	Planning team
Design	Calibration	System identification	Obstacle detection
Fabrication	State estimation	Tracking control	Cost / constraint definition
Integration	Visual perception	Control API development	Motion planning
Modeling	3D perception		Mission planning

Phase III: Integration and evaluation

System integration team	Product team
System integration	User interface development
Logger development	User interface testing
Debugging tool development (visualization, metrics, etc)
Data gathering, machine learning
Iterative development and tuning

Phase IV: Marketing and deployment

Hardware team	Product team	Sales and marketing
Scaling up fabrication	Product requirement validation	Technical documentation
Design for mass production	Certification	Marketing
Supply chain organization	User acceptance testing	Deployment

Caveats

In reality, development will be continual both within a phase and between phases. Within a phase, there will inevitably be iterative evaluation and design as components are tested and refined, and when interacting components are upgraded. There also will be continual work between phases. (... more specifically, between phases I-III, since most robotics companies never get to a product!) Requirements will change, integration challenges will kick problems back to the component teams, data gathered from integration testing will go back to tuning and machine learning, acceptance testing may require repeating the planning phase, etc. So, even though you might worry as mechanical engineer that your job will no longer be needed after the start of Phase II, in reality you are likely to be called upon throughout the development process.

Moreover, in a later section we will describe the concept of vertical development in which teams are created early in the development process to solve Phase III problems. This is a very good idea, as it can be hard to predict all of the integration problems that will be met. User interface development is also often an afterthought in many engineering projects, but getting early results from user interface testing is another very good idea. The end user might find something confusing, or not so useful, or might even be satisfied with a partial product! Having this information at hand can drastically shape the landscape of development priorities and make the difference between success and failure within the development budget and timeline.

Project management

Project management is a "soft skill" that comes in handy both in industry as well as in academia. When proposing a project, whether in the form of a pitch meeting or a grant proposal, the person who holds the purse strings is not only going to want to hear your idea, but also evidence to support confidence that the project will be managed well. This requires giving a plan about how you will manage four key resources: time, money, people, and existing infrastructure. Now, delivering on your plan also requires day-to-day management skills. All that is written on this topic could fill volumes, but we will touch on a few key points in project management here.

Project planning

The first stage of project management is project planning. In your plan, you should articulate both a vision for what you hope to achieve as well as activities that you hope will get you there. The vision is a broad statement about what you hope to achieve by the end of the project. The activities are specific steps that, if successfully executed, will achieve the vision. Usually these activities are organized around deliverables, milestones, phases, or aims. Regardless of what you call them, it is very important for these activities to be articulated in a way that builds confidence in your approach and begins to organize your team's use of key resources. SMART goals are a helpful tool for articulating these activities. The acronym SMART refers to goals that are:

• Specific: the goal should specify what is to be achieved, who will be responsible to achieve it, and generally how it will be achieved.
• Measurable: there should be a success/fail criterion that specifies when the goal is actually achieved.
• Achievable: the goal should be realistic with the constraints of available resources.
• Relevant: the goal should contribute to the overall vision of the project. If it is not obvious, you should articulate how it contributes.
• Time-bound: provide an achievable and relevant deadline for achieving the goal.

An additional (and often underappreciated) consideration is whether your activities are complete and complementary. Completeness means that the vision will be achieved if each of your activities are finished successfully. Complementary means that your activities build on one another and do not duplicate effort. If you fail to articulate a complete set of activities, your audience is left to hope that your team will somehow figure something out to fill the gaps without requesting additional resources. If you fail to articulate a complementary set of activities, it sounds like you are wasting resources.

Don't underestimate how hard this is; it takes practice, experience, and deep thought to write a good plan. I have seen senior managers and tenured faculty struggle through it, resulting in projects being shuttered and grants being rejected. Some typical pitfalls are poorly articulated metrics or evaluation plans, omission of underappreciated steps (of bridging components, integration efforts, or human-machine interfaces, typically), and under-resourcing or unrealistic resourcing (e.g., we will spend $200,000 on a robot and then hire an engineer who will program the application in 6 months).

Scheduling

Another important part of project planning is scheduling. For small projects you can provide estimates based on your prior experience, but as projects grow in complexity and duration, you will need some tools to help understand how people, time, and equipment are allocated to the tasks that make up your activities. One key tool for planning your project schedule is a Gantt chart. These are fairly straightforward charts with a timeline, broken into periods (e.g., days, weeks, months, quarters), on the X axis and tasks on the Y axis. Task dependencies are indicated with an arrow from the completion of one task to the start of another. This might sound obvious, but it is essential that any task that depends on the completion of another starts later in the timeline!

TODO: Gantt chart example.

Another important part of scheduling is assigning tasks to time periods. In doing so, you ought to estimate how long each task will take assuming a given level of staffing, including the optimistic (best-case), realistic (average case), and pessimistic (worst-case) duration. The optimism of your scheduling should be chosen to correspond with your project sponsor's tolerance of time and budget overruns. Each task with a dependency should be scheduled after the end of the dependent tasks. The time spacing between the end of one task and the beginning of another, if any, is a margin of error that your project may tolerate.

A natural question is how long might the entire project may take? To figure this out, we need to determine the critical path through the schedule.

Critical path: the sequence of dependent tasks in a plan whose total duration is the longest.

The critical path can be determined by setting up the tasks as a weighted directed acyclic graph (DAG) and finding the longest (weighted) path. (Although an algorithm can be used to solve the problem, it is usually easy enough to find the critical path by manual inspection). Any tasks on the critical path must be completed on schedule in order for the project to be completed on time, whereas non-critical tasks are often allowed some margin of error.

Note that as stated in a traditional form, critical path analysis does not take into account resource limits (except for time). Limited resources can significantly affect the schedule of a project. Imagine that everything must be done by one person: there's only so much time in the day, so you cannot execute multiple tasks in parallel unless your effort on each task is less than 100%. To come up with a resource-limited schedule, you will need to ensure that throughout the timeline, each simultaneous task does not exceed the level of your available capacity.

Budgeting and personnel

TODO: Budgeting and personnel: < 10% to hardware

TODO: Personnel assignment %FTE

Handling risk

Every plan has some risks associated with it: the likelihood that everything goes perfectly according to plan is quite low! You may be building some untested technology, the implementation of a component might be harder than it looks, end-user testing may reveal that you haven't planned to implement a necessary feature, personnel might quit, IT infrastructure can crash, your whole country may suffer from geopolitical destabilization... We can't eliminate all sources of risk during project management, but we can mitigate their impact.

The first step to risk management is to anticipate parts of the plan that are especially risky. Steps that involve complex implementation, untested technology, end-user evaluation, and regulator certification typically require some degree of scrutiny. Once sources of risk have been identified, a project manager should consider creative solutions to de-risk the plan.

De-risking is industry jargon for reducing the likelihood or the impact of risk on a plan. There are a few ways to de-risk a plan:

• Create a contingency plan which takes effect if the risky condition is met. For example, if algorithm A isn't performing well by date D, begin implementation of backup algorithm B.
• Implement a parallel backup plan to be conducted in conjunction with the main plain, and switch over if the risky condition is met. The main team will implement algorithm A, the backup team will implement B, and you will decide which one to use at date D.
• Iterative development that makes progress toward project goals using multiple design-prototype-evaluate cycles. After each evaluation, lessons learned should be incorporated into the next prototype.
• Because the likelihood of risk is also hard to estimate in advance, it may be helpful to develop risk evaluation steps into the plan which will help better estimate the likelihood and severity of risks. Unit testing, early system testing, and headroom analysis are various strategies to help evaluate risk during the development process.

Design documents and project tracking

Assuming you have followed these guidelines and won approval, congratulations! Your project now needs to start. The first task is for you to get your team aligned on the specific technical steps that need to be executed to complete a project goal. The first place to start is usually a requirements document, which declares technical objectives in terms of specific, measurable aspects of the deliverable. You will then (or simultaneously) write a design document that outlines the steps and timeframe for achieving those objectives.

TODO: Project tracking with Gantt chart, revisions to projecgt schedule, Kanban boards, Github projects, etc.

Horizontal vs vertical development

When developing a product there will often be teams that focus on specific components, as well as teams that integrate multiple components to fulfill specific system functions. These are, respectively, known as horizontals and verticals. This terminology follows the notion of a "tech stack" with high-level, slow components on top and low-level, fast components on the bottom (see the connection to hierarchical architectures?)

Horizontal development: development that focuses on a technical component.

Vertical development: development that focuses on integrating technical components into an overall system function or behavior.

TODO: figure showing horizontal / vertical matrix

Engineers on a horizontal team will focus on refining a component's performance. For example, an object detection team would be a horizontal one and would focus on improving detection accuracy. They will also work with members of intersecting vertical teams to ensure that their component works to implement the vertical function. These will typically be subject-matter specialists with intimate knowledge of the mechanical, electrical, algorithmic, and/or computational aspects of that component. Their performance metrics will typically involve unit testing.

Engineers on a vertical team will focus on expanding the range of functions of the system, or its operational domain. For example, in an autonomous driving company a lane changing team would be focused on producing high quality driving behavior when the vehicle needs to perform a lane change. They will often have specialists in multiple relevant horizontal teams who will work with those horizontal teams to ensure that the system function can be implemented. For example, lane changing may require specialized agent trajectory prediction and motion planning functions, so working closely with those teams should be a high priority for this vertical. In contrast, an object detection horizontal team may not need to be closely involved, since lane changing does not typically require any different object detection capabilities compared to normal driving. A vertical team's performance metrics will typically involve system testing.

It is a common pitfall, especially in smaller organizations, to assign effort only to horizontal components or only to vertical ones. Without verticals, the effort on components may not be well-targeted to produce the desired functions of the system, which leads to last-minute scrambling as product deadlines grow near. Without horizontals, development is slowed down by a lack of coherence and expertise in technical components. You may end up with a mess of code with multiple implementations of motion planners, object detectors, etc. with different APIs, coding conventions, and quality standards. In a real-world example of this, I participated on a DARPA Robotics Challenge team that was vertically oriented. The competition asked teams to develop a robot to complete 8 search-and-rescue tasks, and the theory was to have a lot of professors working on the same team, each of whom had expertise on each task. My students and I were on the ladder climbing team, another professor's lab would address valve turning, another's would address driving, etc. As it turns out, the lack of coordination between task subteams was a big handicap. Although we scored quite well on my event during the semifinals, the team as a whole didn't make it to the finals...

Taking technologies to market

A concept that has gained popularity through its development at NASA and then later adopted by the U.S. Department of Defense, the EU, and the Industrial Standards Organization (ISO) is the notion of Technology Readiness Levels (TRLs).

Technical readiness level (TRL): a rating scale from 1-9 designating the maturity level of a piece of technology, ranging from the basic theoretical principles observed (TRL 1) to fully proven deployments in the operational environment (TRL 9).

Usually, university research operates at TRLs 1-4, at which point a technology is validated in a lab environment. The transition from TRL 4-6 is often accomplished in industry or applied research labs. The last stages of maturing a technology from TRL 7-9 involves product development and refinement, and is almost always accomplished in industry or government labs.

Intermediate stages of development, roughly TRL 4-7, are known as the technological Valley of Death. The reason for this is that many promising technologies are mature enough to be demonstrated in the lab, but the amount of investment required to turn them into a reliable product (known as technology translation) is often underestimated. For example, costs for safety certification of medical devices can run into the tens or hundreds of millions of dollars. This phase is also accompanied by a shift in personnel from the original inventors of the early-stage technology to a development team, and this shift may come at a loss of momentum, enthusiasm, technical expertise, or project management expertise. It may be unwise to ask a professor to start a company!

Another serious risk for any translational endeavor is improper product-market fit, since we technology developers are always enthusiastic about our technology, which leads us to have "blinders on" that prevent us from predicting whether the market (i.e., consumers) will appreciate our product. Robotics is especially susceptible to this kind of failure. The remedy to this tendency is to perform early market analysis by speaking to potential consumers, whether this would be factory owners who might purchase an intelligent automation device, or the general public who might buy a home robot. The results may be eye-opening or even damning to your idea. You may get the best results by switching development priorities, e.g., you find that a new factory robot needs to identify items in clear plastic bags. Or, you may realize that your dream is doomed, e.g., you find that the number of acceptable dishes dropped a home robot is less than 1 per month, but your best lab tests place your algorithm at 5 drops per hour. Convincing your market that their needs are irrelevant is the definition of foolishness. You might be able to convince an investor to give you money for your idea, in the long run, your customers will decide whether your business succeeds or not!

Unit and system testing

System testing (aka integration testing) evaluates whether the entire system performs its specified function according to key performance objectives. This is of course the end-goal of developing a system, but the integration process is expensive and takes a very long time. So, system engineering typically involves a large amount of unit testing, which evaluates whether an individual component performs its specified function.

If designed correctly, unit tests help developers align their efforts toward relevant goals, project managers will have a better sense of priorities for allocating effort, and the whole team develops progressively greater confidence that the system will be ready for its ultimate tests. However, unit testing takes time and can even waste development effort if the metrics are not chosen wisely to align with system-level goals, or the test cases are not chosen properly.

Unit testing

To perform unit testing, a developer will

1. Develop test inputs and supposed outputs (including errors).
2. If the component would interact with other components in the system, develop mock implementations of them, e.g., generating dummy data or replaying data from a log.
3. Create a test protocol (runner).
4. Ensure that the actual outputs of the component agree with the supposed outputs.

For many components, we do not have a perfect idea of what the outputs should be. Instead, the developer will seek to measure performance, using the following process:

1. Develop test inputs and a performance metric(s).
2. Develop mocks, if necessary.
3. Create a test protocol (runner).
4. Analyze the metric(s) and report.

Defining good mocks is extremely important in unit testing, and can be quite challenging in robotics. Essentially, an ideal mock would emulate the outputs of any upstream components so that we can predict how our tested component will perform in practice. There are several ways of getting close to this:

• Stubs: Generate (constant, random, or varied) outputs of the same data type and structure that the upstream component produces. Ideally, the data should also have similar values and behavior.
• Replays: Replay logged data from the upstream component.
• Simulation: Use a high-fidelity simulator of the robot that can simulate all of its actuators and sensors. Execute the tested component using the actual code for all upstream and downstream components.
• Faked simulation: Use a simulator that partially simulates the robot's actuators and sensors, but also create fake implementations of upstream system components to produce "omniscient" readings.

Let's take an object trajectory prediction component as an example, which takes the output of an object detector as input and then extrapolates the future trajectories of detected objects. We would like to mock the object detector. For a stub, we could generate some hypothetical detections of an object moving in a straight line with some noise, and verify whether the predictor generates predictions along that line. For a replay, we would simply record the output of the object detector running on some video data. For a simulation, we would run our test by running the simulation and the object detector on the images generated by simulation. Finally, for a faked simulation, we would skip generating images in simulation, and instead build a fake object detector that reads objects directly from the simulation's objects.

Headroom

In addition to analyzing the expected performance of a component under realistic inputs, it is often helpful to analyze the upper limit of performance of a component under "ideal" inputs. This process is known as headroom analysis. The reason why headroom analysis is employed is to help inform development priorities. Suppose component A takes input from component B and has a performance metric M, and we are deciding whether to invest in an alternative implementation A'. However, the inputs to A' would require us to modify component B to B' or add an additional transformation layer C that would process B's outputs. Instead of implementing these changes (and then potentially having to roll them back if A' doesn't work as well as desired), we can first perform headroom analysis by defining mocks for A' simulating an ideal inputs according to a hypothetical implementation of B'. We can also simplify our implementation of A' to avoid challenging or pathological inputs. If the metric result M' in headroom analysis does not improve significantly on M, then it is not worth investing in implementing full versions of A' and B'.

System testing

To perform system testing, a developer will:

1. Establish the performance critera and a way to measure them.
2. Create test environments that are ideally representative of deployed conditions.
3. Run the system multiple times across the test environments.
4. Analyze and report the performance results.

Measuring performance may involve either manual observation or instrumentation of the test environment.

A system metric that is used by management to measure team or project progress is known as a key performance indicator (KPI).

Metrics

There are many performance metrics used in robotic systems, and here we describe some of the most common ones in use.

Hardware metrics

Actuators / robot arms

• Peak torque: the maximum torque that can be momentarily executed
• Stall torque / continuous torque: the torque can be continuously exerted for extended periods of time, i.e., without overheating.
• Repeatability: the amount of Cartesian error between repeated movements to the same target location. Measured when robot is at rest.
• Accuracy: the amount of Cartesian error between the actual robot location and the target location predicted by the robot's kinematic model. Measured when robot is at rest.
• Reach: the maximum possible distance between a pivot point and the robot's standard end effector point (e.g., wrist).
• Workspace volume: the volume of space that can be reached by the robot's standard end effector point (e.g., wrist).
• Load: the maximum load that can be continuously supported anywhere within the robot's workspace.
• Power consumption: the rate at which energy is consumed by the device, usually expressed in watts (W).
• Amperage: the maximum electrical current drawn by the device, usually expressed in amperes (A).
• Ingress protection (IP) rating: the degree to which the exterior housing of a device prevents entry of particles or fluids into its internal structure.
• Backdrivability: the extent to which a joint can be moved by external forces.

Sensors

• Resolution: the number of elements (e.g., laser readings or pixels) reported across a sensor's field of coverage.
• Field of view: the horizontal and vertical angle range covered by a vision sensor.
• Frames per second (FPS)
• Depth range: depth sensors will have a working range in which depth values can be reported.
• Noise level / signal-to-noise ratio
• Drift: for IMUs, how quickly errors are expected to accumulate over time
• Data transfer rate: the bandwidth of the communication channel used to transfer data from the device to the host computer.

Perception metrics

State estimation / SLAM

• Accuracy: the error between the true state and the estimated state. Usually broken into position, orientation, velocity, and angular velocity.
• Log likelihood: if the state estimator reports a probabilistic outcome, the probability distribution is better calibrated if the log likelihood of the true state given the estimated distribution is higher.
• Recovery time: after an estimate is "lost", this measures how quickly will the estimator converge to a sufficient level of accuracy.
• Final state accuracy: the discrepancy between the initial and final estimate after the robot performs a loop leaving from an initial state and then returning to the exact same state. Often used in visual-inertial odometry and SLAM systems.
• Map accuracy: the error between the reconstructed map geometry or landmarks compared to ground truth.

Object detection

• True Positive / False Positive / True Negative / False Negative rates
• Precision: equal to TP / (TP + FP).
• Recall: equal to TP / (TP + FN).
• Intersection over Union (IoU) / Jaccard Index: for a ground truth bounding box $A$ and the predicted bounding box $B$, the value $\frac{|A \cap B|}{|A \cup B|}$. Perfect detection gives IoU = 1, no overlap gives IoU 0.
• Computation time

Segmentation

• Pixel accuracy: the fraction of pixels properly classified as the correct class.
• Intersection over Union (IoU) / Jaccard Index: for a ground truth segment $A$ and the predicted segment $B$, the value $\frac{|A \cap B|}{|A \cup B|}$. Perfect segmentation gives IoU = 1, no overlap gives IoU 0. Also, mean IoU (mIoU) is a common metric for multi-class segmentation, in which average IoU is computed for each class, and then the average is computed across classes. This approach gives more weight to accuracy on rare classes.
• Dice score / F1 coefficient: the value $\frac{2 |A \cap B|}{|A| + |B|}$. Exhibits similar behavior to IoU.
• Chamfer distance: for a ground truth segment $A$ and the predicted segment $B$, the average distance from each point in $A$ to its closest point in $B$, plus the average distance from each point in $B$ to its closest point in $A$. Compared to IoU, has a stronger penalty for predicting pixels that are far away from the true segment. Perfect segmentation has chamfer distance 0, and the larger the chamfer distance, the farther away the segment from ground truth.
• Computation time

Tracking

System identification

Planning metrics

Computation time is a common metric for all planners.

Kinematic path planning

• Path length: the sum of segment lengths in a planned piecewise linear path, which is equivalent to the integral of the norm of the derivative of the path.
• Failure rate: the fraction of test queries fail to produce a valid path. It is usually assumed that each test query has a valid solution.
• Clearance from obstacles: the minimum workspace distance to obstacles amongst all configurations $q$ in the path $y$: $\min_{q\in y} \min_{O \in \mathcal{O}} d(R(q),O)$.
• Note: for randomized algorithms, you may compute path length / failure rate / clearance either for a given time limit or until success (if test cases are known to be feasible).
• Note: for randomized algorithms, it is important to perform multiple runs on the same environment and start and goal configuration to calculate mean / variance of time / path length / clearance between runs.

Kinodynamic path planning (in addition to kinematic path planning metrics)

• Trajectory duration: how long the execution of a planned trajectory would take, in seconds.
• Control saturation: how close controls get to their extremes.

Trajectory optimization

• Iteration count: the number of outer iterations of an iterative optimizer.
• Evaluation count: the number of evaluations of the objective function / constraint functions and their derivatives.
• Infeasibility rate: when an optimizer terminates with an infeasible solutions.
• Local minima rate: when the optimizer converges to a suboptimal solution.
• Objective value: the minimum objective function value attained by the optimizer. If the objective function is a weighted sum of multiple components, it is useful to report the values of each component.
• Constraint margins: the analytical distance to infeasibility for each inequality constraint, i.e., $max(g(x),0)$ for the constraint $g(x) \geq 0$.
• Sensitivity to initial guess: an evaluation of how much metrics are likely to change depending on the initial guess, e.g., determined by random restarts.

Model predictive control (in addition to control metrics)

• Timeouts / time budget overruns: how often and how badly the optimizer exceeds the budgeted computation time.
• Infeasibility rate
• Local minima rate
• Objective value: the objective function of the optimized path.
• Executed costs: sum of running costs obtained during execution over time steps
• Trajectory consistency: how much does the optimized trajectory shift from one time step to the next.

Multi-agent path planning

• Makespan: max path length of each agent.
• Total path length: sum of path lengths for each agent.

Informative path planning / active sensing

Imitation learning

Reinforcement learning

Control metrics

• Control frequency
• Control bandwidth
• Step response
• Overshoot
• Tracking accuracy

System metrics

Industrial robots

• Mean time to failure (MTTF)
• Return on investment (ROI)
• Cost per unit
• Cycle time
• Mean picks per hour

Autonomous vehicles

• Miles per disengagement (MPD) / Miles per intervention (MPI)
• Accidents / close calls
• Manual driving pose deviation

Aggregation methods

Note that for metrics that are collected over time, from many examples, or along many dimensions, they will need to be aggregated in some way to report a single scalar number.

Max error, MAE, MSE, RMSE. Confidence intervals.

Evaluation

Domain

Validity

Thoroughness

In- and out- of distribution

Analysis

Persistent questions: are our tests representative? Are they conclusive? When do we stop testing and start redesigning? How to tell which component was responsible for poor behavior?

Large-team engineering practices

Dunbar's number

Development methodologies

Waterfall: an organizational philosophy that breaks a project into sequential stages with clearly defined development objectives that must be met before proceeding to the next.

Agile: an organizational philosophy that prioritizes frequent changes to adapt to product and customer needs. It deprioritizes systematic long-term planning due to the inability to foresee precise specifications.

Design documents

Software organization

Code is code, right?

You couldn't be more wrong! Organizing code well is the single most important imperative of software engineering. Code must be iteratively debugged, improved upon, and maintained, and so you and others on your team will need to be able to browse files, read code, identify problem areas, and modify significant parts of your code throughout the lifetime of your project.

Here are some tips to help you start organizing your projects better:

• Separation of concerns: maintain separation between code for algorithms, integration wrappers, settings, models, data, logs, executors, tests, and setup. These can be in different files, different folders, and respect naming conventions that help your teammates (and future you) identify the location and purpose of code.
• Descriptive naming: make sure your files, functions, and variables are named in a way that tells the reader most of what they need to know about their meaning. Don't use generic labels like "myFunction3" or "thevar": tell us something about how these should be used. Group code that performs similar functionality into similarly-named files, or place them in the same folder, module, or package. Follow naming conventions to make your code cleaner and avoid confusion amongst your team. For example, in Python methods and variables that are "private", i.e., not useful for the user of a class, should be prefixed with an underscore (). Using this convention tells a reader of your code that only you, the implementer, should bother trying to understand the purpose of this item.
• Dont repeat yourself (DRY). This is the principle that exactly one version of a function, parameter, or piece of data should exist in your codebase. If you feel the urge to copy your code into a new file (like "my_function_version2.py") to change a parameter, STOP YOURSELF! Instead, make it that parameter a function parameter, or a configuration variable. Let the caller of the function decide which version to use. In some ways, DRY is the starting point to all software engineering. Well-organized projects will have minimal duplication of code. However, in some complex cases it makes more sense to break DRY to duplicate-and-modify. The main reasons to duplicate-and-modify would be that the number of configuration parameters is growing too unwieldy (e.g., a function with 10+ parameters) or that a unified function would be so complex that maintenance would be a nightmare.
• Hard-coding vs soft-coding. Ah, the embarrassing practice of hard coding... we don't usually admit it, but we all do it from time to time. This is generally considered poor practice if you have any intent of sharing your code. It is far preferable to use "soft-coding" practices, such as configuration files, command-line arguments, or parameter servers. If you absolutely must hard-code anything, please, please put these values as constants at the top of your file, with a descriptive name and a comment about how they are used.
• Keep the codebase up-to-date. Remove old code when you have a better alternative, don't just comment it out. We don't want to see things like myfunction-v2.py, myfunction-v2last.py, myfunction-v2lastlast.py. Use Git or some other versioning system to keep track of old versions.
• Document your code. Use comments, readme files, documentation systems, or design documents to help explain how your code can be used and how it works. The very act of explaining your code pushes you through a mental exercise that leads you to more logical and organized decisions. To write well-organized code, you must shift your mental paradigm from "I need to get this code to work" to "how does anyone else use this code?"
• Convention vs configuration. Any significant project will develop components that accept many parameters. Any user of those components will need to specify those parameters somehow, and it takes time to figure out how the parameters affect functioning. There are two design philosophies about how this should be done. The convention-over-configuration philosophy is to provide "sensible defaults" and only require the user to configure a parameter explicitly if they desire less-common functionality. The configuration-over-convention philosophy asks the user to explicitly set parameters rather than accept defaults. The first philosophy argues that convention facilitates rapid prototyping and lessens a novice user's cognitive load, since they do not need to understand the breadth and nuances of functionality provided by the component. The second philosophy argues that implicit parameter setting runs the risk of the user accepting a mistaken default setting, because it encourages lazy thinking. In my experience, robotics is better suited for the convention-over-configuration approach because our systems are so complex that it is nearly impossible for a single person to understand every nuance of every component. When writing your component with a convention-over-configuration philosophy in mind, it is important to follow the "principle of least astonishment", which is that by default your component should behave how most users would expect it to behave rather than be surprised.

Levels of component maturity

D level: "Code scraps"

• Looks like: Messy scripts and notebooks
• Purpose: Testing, rapid prototyping
• Seen by: only you, at this moment in time. Most importantly, you’re not planning to revisit it months from now.
• Code style: you may use whatever style you want. Go ahead and name things badly, others won’t see it.
• Packaging: none. Maybe you throw this into Dropbox / Box when you are done. Or it could be placed in your personal Github account, later to gather dust. Do not push this type of code to your team's project repository.
• Interoperability: None. Dependencies are system dependent.
• Settings: Hardcode things. Delete and replace when you want to change a setting.

C level: Research code

• Looks like: Partially organized code that fulfills a specified, interpretable function. File names and variable names are meaningful, the basics are explained in a readme.
• Purpose: Gathering together useful scraps to make them reusable for yourself and others. To make a checkpoint of your code, e.g., for a big demo or upon paper submission to encourage replicable research.
• Seen by: future you and close colleagues. Someone might need to get a hold of you to learn how to use it.
• Code style: units are organized into meaningful file structures, classes, and variables. Documentation is present but partial. Don’t Repeat Yourself (DRY) is practiced. Code is separated from data and output.
• Packaging: Ideally, put these into your team's project repository, or a self-contained Github repo if you plan to eventually migrate to B-level code.
• Interoperability: Document dependencies in docstrings or readme, or better yet, include setup scripts / requirements.txt. Simplified and suboptimal communications middleware may be used.
• Settings: Configuration files (ideally) or constants at the top of the file. Example settings might be commented out.

B level: Legitimate module

• Looks like: A documented, reusable module that is not embarrassing.
• Purpose: public code releases, releases to collaborating organizations, or to add longevity to your work.
• Seen by: multiple colleagues, some of whom might be using your code after you leave the organization.
• Code style: conformant to typical style guidelines. Organization is solid, DRY is practiced. Module code, tests, data, and settings are separated.
• Packaging: a self-contained Github repo if you want to release it to the public. May work with packaging tools, e.g., pip install.
• Interoperability: communication between parts is documented well. For communication middleware, use best practices in the domain / system on which you are working, e.g., ROS, Google Protocol Buffers, AJAX, etc. System requirements and dependencies are specified in installation instructions (readme) and/or requirements.txt or a setup script.
• Settings: configuration files or documented command-line arguments. Examples or tutorials should be provided.

A level: Maintained package

• Looks like: a high-quality package
• Purpose: public code releases
• Seen by: the world
• Code style: conformant to typical style guidelines. High-quality documentation and tutorials with images and examples.
• Packaging: a self-contained Github repo with continual integration, tests, maintainers, etc. Should work with pip install.
• Interoperability: uses best practices
• Settings: configuration files or documented command-line arguments. Examples or tutorials should be provided.

Software management skills

Github

Branching

Pull requests

Code review

Continual integration

Versioning

Summary

Key takeaways

TODO

Tips

• Budget time and effort to help everyone on your team understand the inputs and outputs of each others' components. Document input/output identifiers, types, rates, coordinate conventions, and units. Make sure to do this at design time, during code reviews, during refactoring, and onboarding new members.
• Make sure that your designs are complete and that personnel are assigned to each component. Avoid magical thinking, e.g., "the robot will figure X out" or that "component Y will be done in Z months".
• Acknowledge errors, uncertainty, and nonstandard user interactions early in the design phase. Imagine the chaos if airlines only let people book tickets, but never accounted for mistakes, flight delays, flight cancellations, maintenance schedules, mechanical failures...
• Prioritize development time on true bottlenecks. Don't let teammates spend too much time on pet projects or reinventing the wheel. "Premature optimization is the root of all evil" -- Donald Knuth.
• System integration middleware is the language by which your team communicates. Pick a package early and stick with it.
• If it ain't broke, don't fix it. Upgrades are dangerous and should only be performed by a trained professional. Robots are designed to work for decades; I've had robots running on Windows XP in the 2010s.
• Making parameters accessible and well-documented is tedious work, but hard-coding is downright evil.
• Develop a single "source of truth" for your system that describes your robot setup, knowledge about the environment, and system state. Without it, errors will be very hard to find.
• Get input from end users before you start. Billions have been wasted by starting projects without a plan to take a product to market.

Glossary

General

• Metric: any quantifiable measure of performance of any thing. Can be continuous, discrete, or binary (e.g., success achieved / not). The subject can be a tech component, a product, a team, or an individual.
• KPI (key performance indicator): a quantifiable measure of a team’s performance, i.e., a human metric. Used by management.
• OKR (objectives and key results): used by Google. Similar to KPI, but prioritizes broad goals rather than quantifiable metrics.
• Headroom: hypothetical upper limit of a technical component’s performance under ideal circumstances and with an ideal implementation. Headroom analysis simulates those conditions, and can help decide how much to invest in a particular method or implementation.
• Development velocity / developer velocity: how "productive" a developer can be over a given timeframe.
• [X]ops (e.g., devops, secops, MLops): teams whose goals are aligned to aid in X velocity, e.g., by providing tools, frameworks, guides, etc.
• Critical path: the sequence of dependent activities in a project plan defining the minimum possible time to complete a project
• Waterfall: an organizational philosophy that breaks a project into sequential stages with clearly defined development objectives that must be met before proceeding to the next.
• Agile: an organizational philosophy that prioritizes frequent changes to adapt to product and customer needs. It deprioritizes systematic long-term planning due to the inability to foresee precise specifications.

Engineering management

• Stakeholder: anyone who has an interest in a project, both internal and external to the organization.
• Technical readiness level (TRL): originating in NASA, a scale from 1-9 defining the maturity of a technology in development.
• Valley of Death: the transition from lab demonstration to a product prototype that often acts as the "killer" of promising technologies.
• Tribal knowledge: information that resides only within human brains on the team rather than in formal documentation, reports, or reference materials.
• De-risk: to take actions to reduce the risks associated with a plan, e.g., to find possible vendors of a component in case the development team fails to meet its development goals, or to develop a fallback implementation that would deliver partial functionality in case an ambitious approach fails.
• Technical debt / tech debt: suboptimal style, structure, or functionality that is introduced when developers take shortcuts in an attempt to make deadlines. Tech debt usually appears as sloppy, badly organized, or non-extensible code, and the debt must be "repaid" later by undoing those introductions.
• SMART goal: a set of principles for writing good milestones and deliverables during project planning. Stands for Specific, Measurable, Achievable, Realistic, and Time-bound.
• % FTE (full-time equivalent): The percentage of effort that one developer devotes to a particular task.

Engineering management pitfalls

• Peter principle: people in a hierarchy rise to the level of incompetence
• Dilbert principle: the most incompetent people in an organization are promoted to management to minimize harm to productivity
• Bike-shed effect / law of triviality: people in an organization commonly give undue attention to relatively trivial issues
• Hofstadter's Law: "It always takes longer than you expect, even when you take into account Hofstadter's Law." Also see optimism bias, planning fallacy
• 90-90 rule: "The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time."
• Student syndrome: planned procrastination, because an impending deadline induces the proper amount of urgency.
• NIH (not invented here) syndrome: a tendency in organizations to avoid using products, software, or knowledge that was derived from outside the organization. Can have legitimate reasons (e.g., licensing restrictions, compatibility) but can also waste time.

Software engineering

• Abstraction:
• Mental model: A person's understanding of the function and inner workings of an abstraction (whether such asn understanding correct or incorrect).
• Toolchain: a sequence of programs designed to accomplish a complex development function
• Unit testing: testing a component of a product to ensure it behaves as expected and/or to gather metrics.
• System testing / Integration testing: testing a whole product to ensure it behaves as expected and/or to gather metrics.
• Regression testing: verifying that new changes to software do not break old functionality, e.g., by introducing new bugs or changing behavior.
• Continual integration: a methodology and toolchains for automatically verifying that a complex software product functions (e.g., compiles, regression tests pass) as desired. Such toolchains are run upon each push.
• Full-stack developer: a developer whose expertise bridges multiple components rather than specializing in a single component.

Software engineering pitfalls

• Software / code rot: code losing performance or functionality over long periods of time due to the environment changing around it, e.g., system or library upgrades
• Leaky abstraction: A (claimed) abstraction of a component that does not fully describe the component's true behavior, such as side-effects.

Exercises

1. TODO: system diagram complexity
2. TODO: reliability
3. TODO: mistakes in Gantt charts
4. TODO: critical path identification
5. TODO: designing a unit test, choosing metrics, test cases
6. TODO: identifying an inappropriate metric
7. TODO: identifying an inappropriate test distribution
8. TODO: poor coding practices: bad naming, hard-coding
9. TODO: poor coding practices: DRY