At Agile 2007, Mary Poppendieck gave a talk entitled 'The Role of Leadership'. Mary started out with a little history going back to 1911 and the book 'Principles of Scientific Management' by Fredrick Winslow Taylor. The book came out about the same time as the 1st Ford assembly line of 1913 and addressed how to make workers more efficient. Mary totally disagrees with the book. Its premise is that workers will do as little as possible, do not care about quality and are not smart enough to know the best way to do the job. To improve efficiency, experts need to define the best way to do a job by breaking down the job into parts and finding the best way to do each part. Workers get paid extra to follow the method determined by the expert. The workers are very happy because they get a higher pay and the employers are very happy because they get higher profit. A lot of these early management thoughts have made their way into current western management practices.
Next Mary covers Charles Allen and his 4 step method of industrial training (Preparation, Presentation, Application, and Testing). To train, get ready and figure out the job, present it and have them do it then do some testing to make sure they know how to do it and follow up on a regular basis until the workers get it. If the learner hasn't learned then the teacher hasn't taught. On job training is the best. Let's train supervisors and teach them how to train the people working for them and cascade it down. This method was tested in the war and proved successful.
20 years later (1940), there was another war and Training Within Industry (TWI) was used. It adapted Allen's approach to take an incredibly inexperienced workforce and train them. The idea was to train line supervisors 1st by teaching them on job instruction (how to train), job methods (how to improve), and relations (how to treat people/solve problems). The method resulted in impressive productivity, but was abandoned in the US after the war ended. However, it got exported to Japan to help rebuild the economy.
1. Knowledge of work: Have to know how to do the job.
3. Skill in instructing: Have to be skillful instructors to pass knowledge onto others.
1. Meetings are repetitive: They are all the same. If the same topic is being discussed over and over again, it means that decisions from last time are not being tracked.
2. The same people do all the talking: This leads to others thinking why am I here?
3. Subjects are beaten to death again again: Decisions need to be honored.
4. We come to decisions just to get the meeting over.
5. I don't have time to code because I am in too many meeting: Notion is that if I am not coding I am not working. In Agile, part of your job is not to code.
6. We have too many people in our meeting: The meeting should include the necessary people needed to get to yes or make a commitment. Having too many people is a sign of distrust and a need for control. These people are standing in the way of commitments.
7. We have too few people in our meeting: Missing product owner, customer representative, tester are too busy. Database developers cannot attend meeting. If all these members are important to decision making they should be in the meeting.
8. With constant stream of meetings, we are treated like machinery not people.
9. Our demos and reviews never bring about anything new: These meetings are not for acceptance. Acceptance should have occurred before hand. The demo is to demonstrate progress. It's an opportunity to act like a team. What are we doing and where are we heading?
10. All decisions were made outside of the meeting. We just had the meeting to be told what we are doing and what to agree to: This indicates that there is still a lot of control going on. Technical architect is deciding what plan should be and what estimates are. Moving from command and control to collaboration is required.
At QCON London 2007, Werner Vogels gave a talk about Availability & Consistency and how the CAP theorem ruins it all. Werner defines scalability by as adding resources and getting a performance improvement proportional to the resources added. Also, if you add resources to improve redundancy, it must not hurt performance. A scalable service is resilient, becomes more cost effective when it grows, is capable of handling heterogeneity (as tech improves, architecture needs to improve), and is operationally efficient (# people necessary to run a node needs to go down as you scale up).
· Decentralize: avoid single point of failure. No centralized components.
· Asynchrony: Make progress under all circumstances even if some parts are not working. Work locally and not worry about the rest.
· Autonomy: Each node should be able to make decision purely based on local state.
· Controlled concurrency: Reduce concurrency as much as possible. Funnel things through single points, change data design to avoid fancy locking.
· Controlled parallelism: Control traffic going to each node so that there is capacity both in CPU and I/O left to do other tasks.
· Decompose into small well-understood building blocks. Same with teams. If it takes more than 10 people then the team is too big (2 pizza team). Knowledge gets shared automatically. If larger, need to have meetings.
· Symmetry: All nodes should do exactly the same thing.
Mary Poppendieck covers Lean Software Development at Agile 2006. She mentions that Lean came into being in 1990 based on book 'The Machine that Changed the World'. The book compares and shows the advantages of Japanese auto manufacturing vs. USA manufacturing. Most of the Lean practices trace their history back to Toyota. Originally it used to be Just In Time (JIT) and then later it became Lean. It's used it manufacturing, operations and supply chain. We can't apply it directly to software development, but we can use the Lean principles and look closed at Lean practices in product development. These principles end up deriving the agile practices.
Next Mary discusses some fundamental principles of Lean. She starts with eliminating waste. The trick here is to decide what waste it. She states that it is the customer's view of waste that matters. MUDA is Japanese for waste or anything that does not add customer value.
Then Mary mentions that Testing is not for finding defects. In lean the job of testing is to prevent defects. If you have a quality process, you are building quality into the code. If you routinely have test and fix cycles, it means you are testing too late. Your process is defective. Final verification should be used to verify that something is working as expected and not for debugging. Defects are a management problem. We should look at defects as caused by a system which allows defects instead of just caused by developers.
A fundamental principle of Lean is to respect people. Toyota's success was based on their ability to harness intellect of ordinary employees.
Lean looks at the whole picture. Software by itself is useless. It needs to be embedded in a business process, hardware, or an activity to become useful. The goal of software development is to support the development of a complete product (a process that helps customers get job done). The team should include business people not just developers.
Next Mary covers the process of iterative development. She also mentions that if churn or requirement change is as high then you are writing requirements too soon. You should move requirements gathering later into the process.
1. Standards: Architecture, Conventions, Tools. These should constantly be challenged and changed.
2. Continuous improvement: Improve the process, refactor the code.
4. Frequent deployment. Small releases, automated deployment, automated installation.
Jean Tabaka gave an overview of DSDM at Agile 2006. Jean begins by giving us some background on DSDM which stands for dynamic systems development method. DSDM has 9 principles. There is nothing remarkable about them other than that business people sat together and agreed on them.
9. Collaboration and cooperation between all stakeholders is essential.
Collaboration and cooperation is one area of differentiation of DSDM. DSDM is very specific about having very clear stakeholder relationships that are explicitly brought in. This is really built into the method and it is not just be hopeful that stakeholders will like it.
Jean then presents the '3 pizzas and the cheese' picture which represents the phases of DSDM. It has phases beyond development. There is the pre-project phase which is a feasibility study and a business study that addresses thing like how a project gets started and where the funding is coming from. Then there is Functional Model iterations, Design and build iterations, and Implementation iterations. Finally there is the post project phase which addresses things like what happens after deployment.
Next, Jean talks about the core agile components of DSDM. They include the 9 principles, the fixed time box, the collaborative and facilitated workshops, and the Prioritized requirement list or PRL. Like other agile methods, in DSDM the iron triangle is flipper over. Time and resources are fixed. Requirements are managed through prioritization and inspection and adaptation. Requirements are baselined at a high level usually using MoSCoW technique (Must have, Should have, Could have, Would like to have it) and delivery is between 2 and 6 weeks.
Other aspects of DSDM include being founded by business owners as opposed to software developers or engineers. It has specific integration points with business processes. It is phase driven. It is consortium-based which gives access to continuous growth of the method. It has licensing fees and multiple certifications which seems appealing to large enterprises.
Some of its benefits include documentation since 1995, continuously evolving, guidance maintained and updated in multiple languages, certification recognized worldwide, and provides a nice enterprise wrapper.
At Agile 2006, Tim Lister gave a talk on Agile Leadership. Tim starts by defining a process. It is what you do not do naturally. Process is interesting when your instinct to do something natural is not optimal. Tim makes an analogy with swimming. Our natural instinct in water is head up and paddle like mad. Afterwards, we learn to sip air, put our head in the water, and come up when we need to. In freestyle competitions, most swimmers choose the crawl technique because it has become a proven process for short distances. A true test of a process is what happens when we are under stress and pressure. What do we do? How do we behave? How to we build? How do we deliver? We should follow the process when things get rough and not only in the good times.
Next Tim defines a leader as someone who by dint of words or deeds influences the behavior of others. A leader inspires, or leads by example. Leadership is not by position. It moves around based on the topic or problem at hand. It will naturally flow to the appropriate person. Position is mostly irrelevant. A PMs jobs is to cultivate leadership.
Tim then moves on to give his own interpretation of the Agile Project Leadership Network's Declaration of interdependence.
1. We increase return on investment by making continuous flow of value our focus.
At least with the time we had, we delivered as many of the most valuable features as we could. All features are accepted, even low value features. However, they just get added to the log and sit there because the client prioritizes items of higher value.
2. We deliver reliable results by engaging customers in frequent interactions and shared ownership.
Deliver reliable results means we make pace visible early. Agile makes work rate or velocity obvious with small increments. You know when you are behind schedule. This eliminates surprises. Engaging customers means we force interaction into a partnership. Knowledge of the customer is always needed. Team needs customer domain expertise.
3. We expect uncertainty and manage for it through iterations, anticipation, and adaptation.
Requirements emerge. They are not gathered. Systems need a gestation period. Anticipation: Why anticipate that the customer always knows what they want. It is rare that we know exactly what we need.
4. We unleash creativity and innovation by recognizing that individuals are the ultimate source of value, and creating an environment where they can make a difference.
This means pushing decisions down and out and making the workplace safe. That is, find the right home for every decision. Don't make a decision that you are not competent to make even though you have the authority to make it. Some decisions need to go down to the developer. Some decisions need to go out to the customer.
Great creativity and innovation is when people start working together on half baked ideas. One person has an idea and others jump in with their own ideas to complement it.
The teams recognize their performance. They don't care what the outside world thinks.
Alistair recommends getting the starter level kit for all the agile styles. He mentions that a methodology is a reflection of the methodology author and people pick a methodology because the style of the author resonates with them.
Alistair then introduces us to Crystal be stating that it aims to be the lightest, least intrusive set of rules that puts a project in the safety zone. Its purpose is to keep people informed and keep them from hurting each other. Its nature is enable people to follow a set of conventions that gets updated regularly. Its philosophy is that people differ in their working styles, projects differ in needs, and software development is communication-intensive, experiment-based, needs lots of feedback in all directions, less is generally better, techniques and technologies change over time, and people learn in class or on the job and not from methodology.
Alistair shares a study he conducted which concluded that people centric methodologies do better than process centric methodology. Methodology is nothing more than a set of conventions that people agree to follow. He recommends adjusting a methodology based on the team and project like having a meeting every month and tuning the methodology based on what was learnt. As the people on the team change, the conventions of the team change. As the project evolves from beginning, middle to end, the strategies and conventions also change.
Alistair then defines the Crystal family of methodologies Clear, Yellow, Orange, and Red. Each one is better suited depending on the project needs.
1. Mindset: Not modeling, but cooperative game where people help each other to succeed. There are 2 goals: The primary is to delivers software, and the secondary is to setup for the next game. These two goals conflicting with each other.
2. Design priorities: Project safety (first and foremost), then development efficiency process habitability (tolerance). Will people use the suggestions.
a. Part of genetic code: Frequent delivery, close communication, reflective improvement.
b. Good ideas: Personal safety (freedom to speak without fear of reprisal), focus (knowing what to develop and having time for it), easy access to expert users, technical environment with frequent integration, automated testing, configuration management.
To get started with crystal, Alistair recommends finding your place on the project chart, picking a frequency of delivery, focusing on the top 3 properties, starting work, and every month doing a reflection workshop and deciding what to change.
Alistair concludes by mentioning that to know if you are doing Crystal look for close communication, see users every week, deliver software every 1 to 3 months, monitor project by quality of communication and moral of community, actively reflect on team quality and working habit.
David Anderson gave a talk on Feature Driven Development at Agile 2006. FDD is highly effective and produces high quality in a faster time to market. It increases productivity and quality and enhances team work. In FDD there is no overtime, no time tracking, no GANTT charts, no task tracking, and no time on task estimates.
FDD methodology involves developing an overall model at a high level and then building a feature list. Planning then proceeds by feature where each feature is designed in detail and then built in small code batches.
Example: Calculate the interest for the bank account. This directly maps to the method name, return value, and class name. David then explains relating a feature to the model. A feature is a method on the domain model.
David then explains relating a feature to the model. A feature is a method on the domain model. There is 1 UML sequence diagram per feature. A feature set is related to a moment-interval on the domain model. All features in a set touch the same pink class on the model. A subject area is related to a chain of moment-interval.
Next David covers the Law of Demeter which ensures loose coupling by stating that classes should only depend on their immediate neighbors. This allows for postponed component definition and making decision at the last responsible moment. This enables componentizations of service orientation which allows building one big code base and breaking into components later and not having to refactor because interfaces between components are defined as the method signature on the classes.
Components are color coded pink, yellow, green and blue. They are re-usable enterprise components. The pinks and yellows are re-usable across multiple greens (the core enterprise components). The greens and blues are re-usable across discrete Enterprise Applications modeled as sequence of pinks.
FDD includes the concept of class or code ownership. This ensures that a class is not doing something it is not supposed to do from a design perspective. Classes are consistent and the api is concise. It also fosters collaboration and team work because if a feature is messaging a class owned by someone else, you need to work with that person.
David then compares bananas to requirement documents. He stresses that they are both perishable and thus cycle time is very important. He introduces CPW( Chief programmer work package) which is a collection or batch of features which can logically be grouped for development simultaneously and can be delivered within 2 weeks or less (each feature must be less than 2 weeks, each CPW must be less than 2 weeks). There is no timebox iteration in FDD. It is however long it takes, but must be less than 2 weeks.
Next feature teams are defined as dynamically formed per feature under guidance of chief programmer. All owners of relevant code are part of the team. This emphasizes team work and ensures that multiple minds collaborate on design and development.
David then provides some productivity statistics of using FDD and briefly covers configuration management in FDD using promotions groups instead of branch and merge techniques. Finally David gives some examples of dashboards used on his projects.
Scott Ambler gave a talk entitled "The Role of Testing and QA in the Agile Software Development" at TASSQ. The talk however covered many topics and was not necessarily geared to testing and QA.
Scott starts by giving an overview of agile software development. Agile development involves open communication, enhanced collaboration, and embraces change. It results in the creation of working systems that meet the changing needs of stakeholders and produces high quality software with the highest value possible for the stakeholder's investment.
Agile values individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan. There are 12 principles in Agile which Scott briefly goes over.
1. Many people confuse agility with code and fix (hacking). Many hackers claim to be agile. These are not agilest. Do they have a test bed? Do they know the stakeholders?
2. Agilist don't model. Agilist don't document. Not true. Agile documentation and modeling is done just in time and avoids unnecessary documentations.
3. ISO/CMM/6sigma therefore we cannot be agile. These presents some challenges' but it is still possible to be agile even in these types of organizations.
4. Agility does not scale. This has been proven to be false. Example is the Eclipse project.
5. Agilist ignore Enterprise concerns. This area is still not well addressed by the agile community.
1. Regular deployment of working software: The only measure of success is delivering working software.
2. Pair programming: This is a hard sell to management, but the data shows that it increases productivity. People learn from each other quickly. Forces programmers to code. Reduces down time (facebook, email, browsing, etc…).
3. Active stakeholder participation: They provide information and make decisions in a timely manner.
5. Refactoring – Small changes to code that improves design and retains the behavioral semantics of code. Enables evolving development assets in a controlled manner, enabling design to remain high quality.
6. Database refactoring – simple change to database schema that preserves the behavioral semantics and the informational semantics.
7. Test driven development. Write test 1st, write code to pass test.
8. Continuous regression testing – Whenever someone checks code in.
9. Test as primary artifacts – acceptance tests are considered to be primary requirements artifacts and unit tests are considered to be detailed design artifacts.
1. Do the simplest thing that can possibly work. Simple does not mean easy, quick or stupid. Simple means it is simple to use. Do the simple thing well before moving into the clever step.
2. Push decisions to last responsible moment. Always ask do I really need to make this decision now or can I put it off for later. Is a decision reversible? If it is then it is not a big deal. But if it is not, postpone it until the latest possible. Early up front design can lead to surprising results. Let the design come on its own with each iteration.
3. Domain Driven Design is natural home for OO. OO is about creating a software simulation of the problem we are trying to solve. The domain model should not know about the presentation layer or the data persistence layer. The domain model will change. Try to get a simulation as clean as possible without technical dependencies. Programmers use the same language as the business person. If the domain model is complex, design the simplest representation possible to get the job done and add to it as more requirements come along. Every time circumstances change and invalidate your model, fix the model. You should be able to articulate the architecture and be able to say this does not fit anymore.
4. Use encapsulation. Ensure classes keep secrets, use dependency injection.
5. Using Test Driven Development drives better design by making it more modular. You benefit from the solution as well as the testing. Having an automated unit tests and acceptance test enables change even in the database schema.
6. Mitigate risks by starting out on parallel path to test out different alternatives.
7. Have basic design beliefs like refactoring, design patterns, design principles, MVC, isolating db from model, focusing on model. Following these believes might not be the quickest way but it is the cleanest.
At JavaOne Conference, Ernie Svehla gave a talk about JEE class loading. Ernie explains that classes can be loaded implicitly as a result of a reference, instantiation or inheritance or explicitly by for example a call to Class.forName(). A class is defined by its name and its associated class loaded. There are 2 key class loading concepts.
2. Delegation: defer a request to load the class to a parent before attempting to load the class myself.
Core class loaders include the bootstrap which load the java.lang.* files from rt.jar, the extension which loads jar files contained in jre/lib/ext, and system or application which loads classes on the class path and shared jars. This creates a hierarchy of class loaders. A user class calls its class loader which delegates to its parent (system). If system cannot find the class, it delegates to its parents and so on (system->extension->bootstrap). If class is still not found, the user class loader then tries to load the class or throws an exception.
Different JEE app servers have different hierarchies and algorithms for loading classes. EJBs, WARS, WEB-INF, manifest, etc. Depending on where classes are, an app might need to be restarted or it can simply be a hot deploy.
Ernie then recommends some packaging guidelines. He recommends leveraging EARs. They should be self contained, include all dependent jars but minimize dependency on core environment like jdbc jars. He recommends packaging common jars needed by wars with ejb jars and to allow wars to be self contained.
Next Ernie covers different application servers and explains how each implements class loading. After that, Ernie gives some examples of the most common errors due to class loading.
4. ClassCirculatoryError: Circulatory dependency or a class and one of its dependents are both dependent on a third class and different class loaders are used to load that 3rd class.
5. Class path issues: improper url specification when using URLClassLoader. url ending with slash references a directory. If no slash, it is referring to a jar file with the name of directory.
6. Class visibility issues: A class cannot be seen due to the class loader hierarchy.
Then Ernie explains how to create a custom class loader by overriding the loadClass() method associated with java.lang.ClassLoader.
Ernie wraps up reminding us that we need to understand how java class loading works; we need to understand how specific JEE servers works and we need to pay attention to how classes are packaged.
Agile Restrospectives – Making Good Teams Great!
Esther Derby and Diana Larsen gave a presentation at Google Tech Talk about agile retrospectives. They explained how retrospectives are tied to the agile practices of inspect and adapt. It fits into the feedback cycle, looking at what you are doing, see if it is working or whether you need to do some changes and try things in a different way. Retrospectives address effectiveness of methods, engineering practices and team work.
1. Set the stage: get everybody's head in the game.
2. Gather data: what happened and how we responded.
3. Generate insights: make meaning of the data.
4. Decide what to do: same, differently, or try new things.
Retrospectives are ongoing and a linked process that flows naturally at the end of each iteration and into the planning part of next iteration. People think that they don't have time for all of this, but these 5 stages can be done in about an hour. They recommend that you don't skip these stages as you might compromise the results.
1. Set the stage: Specify a goal on what we are going to look at and an agreement on how we will work together.
2. Gather Data: Think together as a group and look back over the iteration. Create a timeline and put up events that are important. Try to have a common understanding of the what went on and get a fuller picture of what happened. Also add an emotional graph tracking energy (high, medium, low) along with the events (avoid using the word 'feeling').