tag:blogger.com,1999:blog-56548955267453831892024-03-29T05:11:46.221-04:00AgileJazz - Russ Wangler's BlogAgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.comBlogger20125tag:blogger.com,1999:blog-5654895526745383189.post-70998819285804005022022-07-01T11:52:00.002-04:002022-08-11T13:24:05.342-04:00Machine Learning: what it is, what it's not, and links to go deeper<p>Cassie Kozyrkov is an Artificial Intelligence (AI) / Machine Learning (ML) expert and evangelist at Google. She published a set of training sessions called Making Friends with Machine Learning on YouTube. They are excellent, all six hours of them. To dive deep, watch them.</p><p>This article is a summary of her introduction to Machine Learning presentation. </p><p>To get a flavor for her style, lets contrast Wikipedia’s definition of ML with hers. “Machine learning is a field of inquiry devoted to understanding and building methods that ‘learn,’ that is, methods that leverage data to improve performance on some set of tasks.” Vs. “Thing labeling with examples and truth-finding.” She expresses complexity with simplicity.</p><p>Machine learning is an approach to having computers make an enormous number of small decisions.</p><p></p><ul style="text-align: left;"><li>It is fundamentally different from traditional computer programming</li><li>Artificial intelligence succeeds at very complicated tasks that programmers can’t write instructions for by hand</li><li>Think of it as automating the ineffable</li></ul>Let’s compare the two approaches.<p></p><p>A traditional computer program accepts information input and processes it via statements executed in sequential, conditional, or iterative order. The program’s code is a human readable collection of statements, structures, and algorithms that automate a process. More simply, it is a human readable recipe for solving a problem.</p><p>In contrast, machine learning uses a lot of raw data and sophisticated math to generate algorithms. The implementation details of all of this are beyond human comprehension at anything but an abstract or theoretical level. They are beyond our comprehension because of the sheer volume of equations and because of their use of dimensions beyond length, width, depth, and time. We humans cannot practically problem solve in over four dimensions. But with computer automation, our math can.</p><p>Data Scientists and Engineers don’t code the generative algorithms, they select them from a finite number invented by researchers, for example Neural Networks, Random Forests, or Logistic Regression. They feed the generative algorithms mountains of data and consume enormous amounts of compute resources to generate the Machine Learning algorithms, called models, that are the computer programs they “train.”</p><p>Data Scientists and engineers train their Machine Learning models by providing huge data sets of labeled examples to the researchers’ generative algorithms. Here’s the general structure of the data:</p><p></p><ol style="text-align: left;"><li>An “instance” is an example a.k.a. an observation; a row in a spreadsheet.</li><li>A “label” is the answer, a.k.a. the target, the output, or the ideal output, for the example.</li><li>A “feature” is something we know about the example a.k.a. a variable, a column of a spreadsheet.</li></ol><p></p><p>The algorithms iterate over the instances, evaluating their features, trying an astronomically high number of variations until they, maybe, learn to discern the correct labels for not only their training data but also for data they have never seen, i.e., they didn’t train on. For example, the ability to recognize the sound of the words “hello there” spoken by 100 million unique voices.</p><p>More formally then, Machine Learning is an approach to making or computing many small decisions that involve algorithmically finding patterns in data and using these to make recipes that deal correctly with brand new data.</p><p>Machine Learning was first realized in code in 1952, and its theoretical birth, the invention of the neural network, occurred in 1949. But it only took off when compute and storage resources became fast enough and large enough to handle the massive processing requirements needed to generate useful models.</p><p>Despite the illusion created by talking to Alexa or Siri, the Machine Learning models we use daily are not conscious or ”alive” in any meaningful sense. Humans are required throughout. We curate the enormous labeled data sets of examples. We select and try many models. We verify the results. Over time even our successful models are imperfect and require additional human intervention to identify emergent mistakes and then regenerate the models to maintain their accuracy. If you are its owner, think of an ML model as a high-interest credit card that is never fully paid back.</p><p>Cassie's view is that the biggest problem in AI is using all the right math to answer the wrong question. “We must ask the right questions and solve the right problems. Machine learning will optimize whatever we give it. Give it the wrong thing and you’re in big trouble. Much like the genie in the lamp, the genie is not the problem; it is the wisher. And the wisher does not have to be malign. They can just be foolish and not think through the likely outcomes of their choice.”</p><p>The type of label needed drives the high level approach to machine learning: classification or prediction. </p><p></p><ol style="text-align: left;"><li>Binary classification: image recognition - cat/not cat.</li><li>Multi-class: image recognition of a cat, dog, or weasel.</li><li>Prediction of a numerical outcome $12.14, 10.67, etc.</li></ol><p></p><p>Types of machine learning:</p><p></p><ol style="text-align: left;"><li>Supervised learning: for any example you give the system, you have the correct label handy. Keywords: labeled data.</li><li>Unsupervised learning is the search for patterns. You have data but no labels. Keywords: data mining and clustering.</li><li>Semi-supervised learning is the blend of supervised and unsupervised where we have some, but not all, data labeled. Keyword: partial guidance.</li><li>Reinforcement learning. Here, the system takes a sequence of actions towards a goal that leads to success or failure. For example, learning to play a game. Keywords: sequence of actions, reward/punishment, delayed feedback, system influences its environment (and inputs).</li></ol><p></p><p>Cassie points out that reinforcement learning is really, really hard. It rarely works, but when it does its magic. Think of self driving cars. Their modeling is based on game algorithms.</p><p>So how do you know if machine learning might be a fit for your problem? We’ve already covered that the problem should be ineffable, meaning it’s not practical for a programmer to write code to solve it. The next question is: can you imagine what sort of decisions or labels would the machine learning system make for you? If you cannot answer that question, then stop. It’s too early. You’ll need to derive insight by analyzing the problem space using descriptive analytics first. </p><p>Which brings us to a basic process description of Data Science:</p><p></p><ol style="text-align: left;"><li>Use descriptive analytics to get inspired and discover a problem machine learning may solve for you. </li><li>Problem in hand, use machine learning to generate a recipe, i.e., use technology to create the model. </li><li>Test the efficacy of the model using statistics. Remember the Genie; decide wisely.</li></ol><p></p><p>Links to Cassie’s videos:</p><ol class="ol1"><li class="li1" style="font-family: Arial; font-size: 14px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><a href="http://bit.ly/mfml_part1"><span class="s1" style="font-kerning: none;">http://bit.ly/mfml_part1</span></a></li><li class="li1" style="font-family: Arial; font-size: 14px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><a href="https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqa3BuQ25XOU5kbzVOeFBMQ1VMVDdrdFRuR2t3UXxBQ3Jtc0tsQzB1NDBLbzNBSEU0cmQtN2RxS3l3TGxrblRQdWUwX1lIekd0VFh3Z2t3bUlVaFBsbzRMRU8tUGZyck9jZF91ZzJWTEhLZGd6c3Y0cGJ3LTBUbk1Rd0thLWItUzVvZFpYOHJmWkVGT01hUE1sZUxVMA&q=http%3A%2F%2Fbit.ly%2Fmfml_part2&v=lYWt-aCnE2U"><span class="s1" style="font-kerning: none;">http://bit.ly/mfml_part2</span></a></li><li class="li1" style="font-family: Arial; font-size: 14px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><a href="http://bit.ly/mfml_part3"><span class="s1" style="font-kerning: none;">http://bit.ly/mfml_part3</span></a></li></ol><p class="p1" style="font-family: Arial; font-size: 14px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span class="s1" style="font-kerning: none;">Link to Cassie on Medium: <a href="https://medium.com/@kozyrkov">https://medium.com/@kozyrkov</a></span></p>AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com0tag:blogger.com,1999:blog-5654895526745383189.post-3330068150183656582022-06-16T11:36:00.010-04:002022-07-01T11:47:10.322-04:00Principles, motives, and metrics for software engineering<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0zN6hn141yuYqJd_VXgwD71jzBDZ9BLn9v5Poj4Tbev6uuyETuLWDuPTgPlSPv4BD5vGckzb66NONOVMXWPWuVc4xtP76Cmdbzl_sWJrT2RVsBWvkFqSuyfBOwhBYfnkUjayQy4oyn-pXoqJKc7-50bPmRUC5rtvsOlDSI_dNQFISX2OcV2r-awKkUA/s476/LI%20Metrics%20Cover.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="280" data-original-width="476" height="235" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0zN6hn141yuYqJd_VXgwD71jzBDZ9BLn9v5Poj4Tbev6uuyETuLWDuPTgPlSPv4BD5vGckzb66NONOVMXWPWuVc4xtP76Cmdbzl_sWJrT2RVsBWvkFqSuyfBOwhBYfnkUjayQy4oyn-pXoqJKc7-50bPmRUC5rtvsOlDSI_dNQFISX2OcV2r-awKkUA/w400-h235/LI%20Metrics%20Cover.png" width="400" /></a></div><p>I was recently asked for a point of view on organizational metrics for a new group of Agile Teams at a big software engineering shop. The conversation led to this quick essay to collect my thoughts. Big topic, short essay, so please forgive omissions. </p><h3 style="text-align: left;">Context</h3><p>Business circumstances and organizational philosophy dramatically influence metric and measure choices. At one extreme frequent releases delivered when ready is the best business fit. On the other extreme infrequent, date-driven deliveries addressing external constraints are the appropriate, if uncomfortable fit. While they share the basics, the level of formality, depth, and areas of focus are different. Other considerations abound.</p><p>Another key dimension of metric selection is the type of work. When dealing with garden variety architectural and design questions, the work has a clear structure and common metrics will work well. However, when there are significant architectural and design unknowns, it is hard to predict how long the work will take; thus, common metrics and measures are not as helpful.</p><p>So for this essay, I am assuming:</p><p></p><ul style="text-align: left;"><li>Multiple Agile teams with autonomy matched to competency, or put another way, as much as they can handle.</li><li>Teams own their work end to end, from development through operations.</li><li>There are inter-team functional and technical dependencies.</li><li>Frequent releases are delivered when ready (I won’t go deep into forecast actuals, etc.).</li><li>A traditional sprint-based delivery process (I am not saying that’s best, but it’s popular and easy to speak to in a general-purpose article).</li></ul><p></p><p>Motivations, What Questions are we asking?</p><p></p><ol style="text-align: left;"><li>Are we building the features our customers need, want, and use?</li><li>Are we building technical and functional quality into our features?</li><li>Given the business and market circumstances, is the organization appropriately reliable, scalable, sustainable, and fair?</li><li>Is our delivery process healthy and are our people improving over time?</li></ol><p></p><h3 style="text-align: left;">Principles</h3><p><b>Single metric focus breeds myopia - avoid it:</b> asking people to improve one measure will often cause that measure to appear to improve but at the cost of the overall system. This leads to…</p><p><b>Balance brings order: </b>when you measure something you want to go up, also measure what you don’t want to go down. You want more features faster - what’s happened to quality? Are we courting burnout? It is easy to create a perverse incentive with a manager’s “helpful metric.” </p><p><b>Correlation is not causation:</b> It is hard to “prove” things with data. Not because the data is scarce, though it often is, but because there are frequently many variables to consider. Understanding is a goal of measurement. Yet often the closer you look, the more variables you see and the less clear the relationships may become. So, be warned, if only “a” drives “b” it is likely you are missing “c” and “d,” etc. Be cautious.</p><p><b>Competence deserves autonomy:</b> a competent team should choose how to manage and solve their own problems, including what to measure. Since a manager has problems too, it’s the same for them. As long as the leader can explain why they need it and how it should help, they’re good. Force it without the team’s buy-in, it likely won’t work, anyway.</p><p><b>Wisdom is hard won:</b> Metric targets are seductive. Having no targets comes across as having no goals. Aimlessness. But bad targets are pathological and that’s worse. A competent, productive developer who’s not improving is better than one held to myopic targets they neither want nor value. Unless your aim is a dramatic change, set targets relative to current performance, not an ideal. </p><p>If you demand dramatic change, you better know what you're doing. No pithy advice for that. Good luck.</p><h3 style="text-align: left;">Measures and Metrics</h3><p>Based on the circumstances, the level of formality and utility will vary.</p><h4 style="text-align: left;">Customer Value Measures</h4><p></p><ol style="text-align: left;"><li>Feature use - does our customer use the feature we built?</li><li>Net promoter score - do they love our features enough to recommend them to others?</li></ol><p></p><h4 style="text-align: left;">Delivery (team level)</h4><p>Sprint Metrics - simplistically said: Push work into a time box, set a sprint goal, commit to its completion, and use team velocity to estimate when they’ll be “done” with something. Most argue the commitment motivates the team. Some disagree. </p><h4 style="text-align: left;">Cumulative flow for Sprint-based work</h4><p></p><ul style="text-align: left;"><li>Velocity or points per sprint</li><ul><li>Should be stable, all else being equal. Sometimes all else is not equal, which doesn’t mean something needs “fixed,” but it might mean that. Sorry.</li><li>Using Average Points Per Dev can reduce noise by normalizing for team size/capacity changes week to week. </li></ul><li>Story cycle time</li><ul><li>Time required to complete a development story in a sprint. When average cycle time exceeds 2/3’s of sprint duration, throughput typically goes down as carry over goes up.</li><li>In general, analysis of cycle times through work steps across processes and teams can be very revealing (enables you to optimize the whole). But, that doesn’t just fall out of Cumulative Flow.</li></ul><li>Backlog depth</li><ul><li>Generally indicates functional or technical scope changes</li><li>Doesn’t capture generalized complexity or skill variances that impact estimates</li></ul><li>Work in process per workflow step</li><ul><li>Shows when one step in the workflow is blocking work</li></ul></ul><p></p><h4 style="text-align: left;">Work allocation</h4><p>When one team does it all, end to end, keeping track of where the time goes helps understand what you can commit to in a sprint. Also, products need love beyond making them bigger, so you need to make choices for that.</p><p></p><ul style="text-align: left;"><li>% capacity for new features</li><li>% capacity for bugs/errors</li><li>% capacity for refactoring and developer creative/ learning time</li><li>% production support</li></ul><p></p><h4 style="text-align: left;">Functional Quality</h4><p></p><ul style="text-align: left;"><li>Bugs</li><ul><li>Number escaped to production - defects</li><li>Number found in lower environment - errors</li></ul></ul><p></p><h4 style="text-align: left;">Technical Quality</h4><p></p><ul style="text-align: left;"><li>Code quality</li><ul><li>Unit test coverage %</li><li>Rule violations - security, complexity (cognitive load)... long list from static analysis tools</li><li>Readability - qualitative, but very important over time</li></ul><li>Lower environments (Dev, Test, CI tooling, etc.)</li><ul><li>Uptime/downtime</li><li>Performance SLAs</li></ul></ul><p></p><h4 style="text-align: left;">Dependency awareness</h4><p>Qualitative, exception-based. How well does the team collaborate with other teams and proactively identify and manage dependencies? </p><h4 style="text-align: left;">Team morale</h4><p>Qualitative, could be a conversation, a survey, observation, etc.</p><h4 style="text-align: left;">Mean time to onboard new members / pick up old code</h4><p>Most engineers don’t like to write documentation. Sure, “well-written code is self-documenting” but they also know better. A good test is how long it takes to onboard new members / pick up old code. </p><h4 style="text-align: left;">Estimating accuracy (forecast vs. actual variance)</h4><p></p><ul style="text-align: left;"><li>Top-down planning - annual or multi-year. Typically based on high-level relative sizing.</li><li>Feature level - release or quarterly level. Typically, a mix of relative sizing at a medium level of detail.</li><li>Task/story level - varies by team style. May be relative or absolute (task-based).</li></ul><p></p><h4 style="text-align: left;">Cross-team metrics</h4><p></p><ul style="text-align: left;"><li>API Usability, qualitative - for teams offering APIs how much conversation is needed to consume the API</li><li>Mean time to pick up another team’s code - documentation and readability</li><li>Cross-team dependency violations</li></ul><p></p><p>So, there you have it. Some thoughts on principles, motives, and metrics for software engineering. Comments welcome. Be well. </p>AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com0tag:blogger.com,1999:blog-5654895526745383189.post-1713217056874117922022-06-10T10:02:00.001-04:002022-07-01T11:33:32.171-04:00Coaching and Creating a Culture of Learning<p> </p><div class="separator" style="clear: both; text-align: center;"><img border="0" data-original-height="404" data-original-width="1002" height="161" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3RhpBCZ90Lb46wOaTAq9ZKTF6irIVMaWM3xly5ToGxNlC7vih0FB9V16FSFybyvs2alVVT89FPBUyRbIqVhp-ubCmiPtGGTNe0wsLE42AL5sIK8IZ04mo6m_iCbQ-hhGTVFhXytk_vs5RwW8vMr3PrPefUoiZolJAEzelRE6niSh5SkpIy-6cRNRnUQ/w400-h161/SeeSaw.png" width="400" /></div><p></p><div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;"><span style="font-family: "Times New Roman"; font-size: 11pt; font-weight: 700; white-space: pre;">Coaching is a skill</span></div><span style="font-family: "Times New Roman"; font-size: 14.6667px; white-space: pre;">An ability to help others improve a skill through a combination of</span><br style="font-family: "Times New Roman"; font-size: 14.6667px; white-space: pre;" /><ul style="font-family: "Times New Roman"; font-size: 14.6667px; white-space: pre;"><li><span style="font-family: Times New Roman;">Observing behavior related to a particular skill</span></li><li><span style="font-family: Times New Roman;">Asking questions and listening to understand strengths and opportunities for improvement</span></li><li><span style="font-family: Times New Roman;">Providing support and feedback via a mix of</span></li><ul><li><span style="font-family: Times New Roman;">asking and telling</span></li><li><span style="font-family: Times New Roman;">observing and showing</span></li><li><span style="font-family: Times New Roman;">getting involved and leaving room for independent learning</span></li></ul></ul><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "Times New Roman"; font-size: 11pt; font-weight: 700; white-space: pre;">It is about facilitating learning</span></p><div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;"><span id="docs-internal-guid-2713001b-7fff-d198-e719-60dc6d18b8ab"><span style="font-family: "Times New Roman"; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Mutual respect and purpose enable success anything else courts failure.</span></span></div><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"></p><div class="separator" style="clear: both; text-align: center;"><div style="text-align: left;"><img border="0" data-original-height="359" data-original-width="340" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj73g_YIfEvrYSEKhrldLFCpOzb4Mo4jhJx5UEE8tM4RBH_dLIMxW_rX90lISixsE8rYiVFfzNWQ5clk9x407Io9UnmMrTGCGoHwly5tpVB5azyBeXI9Y-6e-UucOyw4j583ZtmpUUsH8ecPrqIpOj02oBast5S-bNpNz-EhxIA3EwqrCDxUrv0iJrAZQ/w379-h400/Coaching%20Model.png" width="379" /></div><div style="text-align: left;"><b><br /></b></div><div style="text-align: left;"><b>Learners move through stages of competence</b></div></div><p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Times New Roman'; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><span></span></span></p><div class="separator" style="clear: both; text-align: left;"><img border="0" data-original-height="218" data-original-width="320" height="272" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEha08JbNPMg-IreEnXtBAcTiWyzP5PeMFsmUgIVR1VQ-O7tqIDS9Hs1VGtzQdVab7ku4o6YrG65vLBPJhmXVVfqaYCLOYvA0UfhNTxAMKz2v9qpIv0wsmSEeBbxwJOOKxmgexQZolOAMvBtRKEqNvmGQocK2ae8LSZ7k0Ple8LaiNcw3UfZAWvVrbhgNg/w400-h272/Competence.png" width="400" /></div><div><br /></div><div><b>Coaches ask a lot of questions to engage</b></div><div><b><br /></b></div><div><span style="font-family: Times New Roman;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3mthPJMjm_cfYXHJFPqrHtjyDj-jo_jT4YmyI5e9cL5EYNbZ5XxPHPqVX43ru1UuvGmkYiixfVDLO_-q_sSxMlRi0Bo2hQqSXt569I193Y70KHJ6EA2qYvKs54B773sZEcZ0j4ud9b4F7DSBpxXZ7hWhk1hEtjVFyiljZlEDytvzW9MgyV9DxjTPnIQ/s576/Open-Closed.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="347" data-original-width="576" height="241" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3mthPJMjm_cfYXHJFPqrHtjyDj-jo_jT4YmyI5e9cL5EYNbZ5XxPHPqVX43ru1UuvGmkYiixfVDLO_-q_sSxMlRi0Bo2hQqSXt569I193Y70KHJ6EA2qYvKs54B773sZEcZ0j4ud9b4F7DSBpxXZ7hWhk1hEtjVFyiljZlEDytvzW9MgyV9DxjTPnIQ/w400-h241/Open-Closed.png" width="400" /></a></div><br /><b><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /></b></span></span><span id="docs-internal-guid-93afab11-7fff-9a01-1fcb-57940a79f52e"><span style="font-family: "Times New Roman"; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"><br /></span></span></div><div><br /></div><div><b>Coaches give good feedback, a model</b></div><div><span style="font-family: Times New Roman;"><span style="white-space: pre-wrap;"><b><div class="separator" style="clear: both; text-align: left;"><img border="0" data-original-height="403" data-original-width="594" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi63x0DqCCY4PiE4hiOYB5oqFlJ-qlCWoycipOucM9IFQjCGKFt_qEZWtXiMdOohF_BLRyQwgW6jugfhRBdDeVB4rcWue4MNPep821EBBTDQ1VveLexBEpOAo_om6yRWgQrj7JpS72Drd2i2NUwRlCdT2v73BbGvkFO80eXQZ6e9equypRe6YBoNvYNiQ/w400-h271/Table.png" width="400" /></div></b></span></span><div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><ul style="text-align: left;"><li>Have a conversation.</li><li>Avoid making judgments or expressing personal feelings about what you have seen.</li><li>Remember, it’s not what you say; it’s how you say it.</li><li>Be supportive and respectful when giving feedback.</li><li>Maintain and enhance self-esteem when giving feedback.</li></ul></div><div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><b>Coaches balance purposeful interactions</b></div><div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRBFHvTsYvBSzAjkUknE7B9ysbQMDUgOx3QxePPiXymyy_QtMQmq3ufz12YUQaXQnnhWnJ1CyTD6YTHMVG-XIJFYy9ougPUX4AhDtQVtbLOSlza34KLKjiNsV3h5pZPR_RfaNs7NebudqFz-zBYSnK7WwHIlWwDO7yON3J_kPNrQM_ELQ3I0pVbemPaQ/s1002/SeeSaw.png" style="clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="404" data-original-width="1002" height="161" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRBFHvTsYvBSzAjkUknE7B9ysbQMDUgOx3QxePPiXymyy_QtMQmq3ufz12YUQaXQnnhWnJ1CyTD6YTHMVG-XIJFYy9ougPUX4AhDtQVtbLOSlza34KLKjiNsV3h5pZPR_RfaNs7NebudqFz-zBYSnK7WwHIlWwDO7yON3J_kPNrQM_ELQ3I0pVbemPaQ/w400-h161/SeeSaw.png" width="400" /></a></div><div><span><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><div>Approach coaching sessions on a case-by-case basis…</div><div>Seek to make themselves obsolete for this skill for this coachee</div><div>Deal effectively with the emotions that come up so that the coachee values the experience (even when it is not fun)…</div><div><b>Avoid the “expert’s mistake”</b> An expert often shares so much information that the learner gets lost. </div><div>An expert may:</div><div><ul style="text-align: left;"><li>Operate from ego (look at how much I know); or</li><li>May feel like they need to share everything they know so the coachee won’t make a bad decision, or simply </li><li>Forget that a learner can only absorb a certain amount of information at a time</li></ul></div><div><b>Create a Culture of Learning, Apply Adult Learning Theory</b></div><div>Malcolm Knowles introduced Adult Learning Theory in 1968. The fundamental pillars are that </div><div><ul style="text-align: left;"><li>Adults want to participate in both the planning and evaluation attached to their instruction.</li><li>Experiences, both good and bad, serve as the backdrop for all learning activities.</li><li>Adults first gravitate towards learning things that are directly relevant to their job or personal life.</li><li>Adult learning centers on problems, not subjects.</li></ul></div><div>Adults generally learn</div><div><ul style="text-align: left;"><li>10% through Traditional methods (reading and lectures to learn concepts and facts)</li><li>20% through Relational methods - learning from others</li><li>70% through Experiential methods - learning on the job</li></ul></div><div>Coaching is exercised in Relational and Experiential methods.</div><div><br /></div><div>Tips:</div><div><ul style="text-align: left;"><li>Quiz students before each traditional learning session so they may ask themselves:</li><ul><li>What do I already know? (before)</li><li>What am I to learn in this session? (when I’m done)</li></ul><li>Quiz students after each traditional learning session to support retention.</li><li>Use “spaced repetition” for facts and concepts that must be retained for long periods of time</li><ul><li>Spaced repetition is an evidence-based learning technique that is usually performed with flashcards. </li><li>Newly introduced and more difficult flashcards are shown more frequently, while older and less difficult flashcards are shown less frequently in order to exploit the psychological spacing effect. The use of spaced repetition has been proven to increase the rate and retention period of learning</li></ul></ul></div><div><b>The Adult Learning Model</b></div><div style="font-family: "Times New Roman"; font-size: 11pt; font-weight: 700; white-space: pre-wrap;"><br /></div></span></span></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixETeTBoeQlNSqYQksASfadeNYGo3IQ76zc5qi8SM13NBiB7G50SlJGdRmC7l2QqrC1ejT8R-LA0tSRjuW-EUtFn1de_0pMHculvAgF4hGt5MChZQq-62TVOxo5hnL3EzvvP7Puqz6XJSegR1aBoWR3ip_hZdCQutzdmAfcJC7odic6pojFCPaMm0MVg/s674/Learning%20Model.png" style="clear: left; float: left; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="528" data-original-width="674" height="314" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixETeTBoeQlNSqYQksASfadeNYGo3IQ76zc5qi8SM13NBiB7G50SlJGdRmC7l2QqrC1ejT8R-LA0tSRjuW-EUtFn1de_0pMHculvAgF4hGt5MChZQq-62TVOxo5hnL3EzvvP7Puqz6XJSegR1aBoWR3ip_hZdCQutzdmAfcJC7odic6pojFCPaMm0MVg/w400-h314/Learning%20Model.png" width="400" /></a></div><br /><span style="background-color: #fefefe; color: #0a0a0a; font-family: 'Times New Roman'; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span><p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;"><span style="background-color: #fefefe; color: #0a0a0a; font-family: 'Times New Roman'; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;"><span style="background-color: #fefefe; color: #0a0a0a; font-family: 'Times New Roman'; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;"><span style="background-color: #fefefe; color: #0a0a0a; font-family: 'Times New Roman'; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;"><span style="background-color: #fefefe; color: #0a0a0a; font-family: 'Times New Roman'; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;"><span style="background-color: #fefefe; color: #0a0a0a; font-family: 'Times New Roman'; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;"><span style="background-color: #fefefe; color: #0a0a0a; font-family: 'Times New Roman'; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;"><span style="background-color: #fefefe; color: #0a0a0a; font-family: 'Times New Roman'; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;"><span style="background-color: #fefefe; color: #0a0a0a; font-family: 'Times New Roman'; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></p><div class="separator" style="clear: both;"><b><br /></b></div><div class="separator" style="clear: both;"><b>Applying the Adult Learning Model</b></div><div class="separator" style="clear: both;">The flow below illustrates a SME, subject matter expert, conducting a set of “brown bag” learning sessions over a number of weeks. This approach has delivered significant and sustainable benefits and builds a culture of learning when it takes root.</div><div class="separator" style="clear: both;"><img border="0" data-original-height="1002" data-original-width="1062" height="378" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQ7zTGsPY2_FGtPZkLG_73qztho4G4WMFC3NyftkuFHImnVVFnU1WuXQ_0kmK2JChKBTgaXAuoSmMkHVMFm-8HgYdzYT2_Uv57tMVAkJ0KbsQUEipGfQ7kiPuQekQKn_qcPKBkFyAak9ay-hyBt8hNNa78GTeNNfHYHM80QNknP-hzMfWHdNqN06dClg/w400-h378/Flow%201.png" width="400" /></div><div><br /></div><div>Here is a more sophisticated illustration based on a coach joining a team to teach a complex skill requiring 3-6 months of daily engagement. </div><p dir="ltr" style="line-height: 1.38; margin-bottom: 16pt; margin-top: 12pt;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVU_UskqgIs7wSynu16gYITX7VVc3LrDEOPRdV656eO_tYhF3EI3GeWwkUJkvi8UyfkVDJMeCY9gnmFyQyALOkZwrQqDEB7iSW1udU8dIKBuwrgPm9w9ThupUNOgKQ74POMzeO1Q9q6re22U6QFDtZkBnwFVfRKAbeD7LtYff0LnHQo6HGAUSSGVvIYQ/s1600/Flow%202.png" style="clear: left; float: left; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="695" data-original-width="1600" height="174" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVU_UskqgIs7wSynu16gYITX7VVc3LrDEOPRdV656eO_tYhF3EI3GeWwkUJkvi8UyfkVDJMeCY9gnmFyQyALOkZwrQqDEB7iSW1udU8dIKBuwrgPm9w9ThupUNOgKQ74POMzeO1Q9q6re22U6QFDtZkBnwFVfRKAbeD7LtYff0LnHQo6HGAUSSGVvIYQ/w400-h174/Flow%202.png" width="400" /></a></div><br /><span style="background-color: #fefefe; color: #0a0a0a; font-family: 'Times New Roman'; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span><p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"></p></div>AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com0tag:blogger.com,1999:blog-5654895526745383189.post-60399082360241158692022-04-21T07:39:00.174-04:002022-07-01T11:35:26.201-04:005 Problems Big Data Architectures Solve<p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTlY-OmjS7o51DwCE5vPYbrT-lR7z8rU_PZpwLKBzxSYfCJs7wBNCXfBRUHriEt0nyJVaYVwvjoTiwODk0kxksAuJySRWFLOovTHbbavDD6AacFXLqGdutWNWaPCClObUzA1_R_Jrup8tAfaieY7NKY_e6cMJcvEva46H8FSNhCuVawSDYThSk_i164w/s5413/Big-Data-Architecture.jpeg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3609" data-original-width="5413" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTlY-OmjS7o51DwCE5vPYbrT-lR7z8rU_PZpwLKBzxSYfCJs7wBNCXfBRUHriEt0nyJVaYVwvjoTiwODk0kxksAuJySRWFLOovTHbbavDD6AacFXLqGdutWNWaPCClObUzA1_R_Jrup8tAfaieY7NKY_e6cMJcvEva46H8FSNhCuVawSDYThSk_i164w/s320/Big-Data-Architecture.jpeg" width="320" /></a></div><span id="docs-internal-guid-156c3ec9-7fff-31ce-4912-997fb69475a3"><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The breadth of Big Data and Analytics technical architecture can seem intimidating. Despite its diversity, though, it solves a handful of problems. It makes trade-offs and surprise, just moves work around to meet its performance goals.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I’ll attempt to describe the key problems and the tradeoffs simply. These basics should help demystify architecture selection and troubleshooting for technology leaders new to the space.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Big Data is well, big. Too big for one machine to process quickly and store reliably. So, we need a lot of machines, working in parallel, and will have to coordinate their activity. How many? In the extreme, thousands. </span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Data may stream in so fast that it is hard to even ingest it without falling behind. With so much distributed data, finding and reading it quickly becomes a problem. Finally, reliable storage requires multiple copies and reliable coordination requires a way to handle coordinator failure.</span></p></span><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">In short, the problems are to:</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">1. Scale write, read, and processing performance</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">2. Parallelize work across nodes, i.e., machines.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">3. Efficiently share data and coordinate activity across nodes.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">4. Reliably process and store data despite one or more node failures.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">5. Meet your business needs for data consistency and availability.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><b>Problem 1: Scale write, read, and processing performance.</b></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><b>Writing</b></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">At scale, simple things like reading and writing become hard. The physical world is unforgiving, so learn your constraints. You can’t have it all. Ultra fast writes typically means slower reads and vice versa. When you can solve for both, accept that you’ll create a new problem. There are no free lunches at scale.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">In a nutshell, store your data in ways that make it easy to do what you need to do with it. And address the consequences. Let’s look at some examples.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">A traditional transactional database needs data consistency, so its relational model limits data redundancy. It needs reasonably fast reads and writes, so it defers some of its disc housekeeping, index and other storage optimization to off hours. Just don’t forget to schedule it!</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Specifically, most Relational Databases record where they store their data on disc in real time using B+ Trees. Each file write causes multiple updates to the B+ Tree and takes time. To avoid more delay, it writes data somewhat haphazardly to disc blocks, slowing reads. The deferred work moves to clean up jobs. </p><div class="separator" style="clear: both; text-align: center;"><span style="border: none; clear: left; display: inline-block; float: left; height: 150px; margin-bottom: 1em; margin-right: 1em; overflow: hidden; width: 208px;"><img height="150" src="https://lh4.googleusercontent.com/T-jp2MYd0n6zw8-DcO65ab4hTV5oKwQd3XyH59wjuU299Y0ODWWzkPJPMoaM19HNJtfbVhTkW4Gg-Vta3qjLwyWAu1vQ7xWHjHAEF6oDT9QquF99SzB1RBMSFMT9oBlR3dr73UvE" style="margin-left: 0px; margin-top: 0px;" width="208" /></span><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-align: left; vertical-align: baseline; white-space: pre;"></span><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-align: left; vertical-align: baseline; white-space: pre;"><span style="border: none; display: inline-block; height: 155px; overflow: hidden; width: 243px;"><img height="155" src="https://lh4.googleusercontent.com/46YfL7qsNA1A1zySZ6vTLjElHXQQv48m9FO2F0hx5-Lyc-OFQqgmeee3CozJHG_vLxW0BH7lX-WNfFdbQKy_9GjwFdmbfgr4vHCLe5rnN8gi27qkrcByq8hQVOXh-z1ndjIU8NbD" style="margin-left: 0px; margin-top: 0px;" width="243" /></span></span></div><div class="separator" style="clear: both; text-align: left;">Source: <u style="color: #1155cc; font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><a href="https://en.wikipedia.org/wiki/B%2B_tree " target="_blank">Wikipedia</a></u></div><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">What if we need faster writes? Well, something has to give. Cassandra, for example, provides faster writes, strong read times, and lightweight transactions. The cost is more data redundancy and less consistency. You also have to agree to less flexible read options — sorts move from query parameter to design decision. But wow, very fast.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">To do this, Cassandra, and others, e.g. Kafka, use a Log Structured Merge Tree. Writes are batched based on available RAM (memtables) and appended to a log (SSTables). Deletes and modifications are appended to the log rather than applied to the underlying data store. A compaction process handles data storage changes and optimizations, again in batches.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><img height="186" src="https://lh5.googleusercontent.com/imrLuuYddwiL7GEuiW447hAXA6ws2FLczQ6qecnTdeScabJ3Jy2trNUG_6Dko-v6OSjJTd-ehXVvAD23dYo8Gbiqb0LyCVtp2wTQ_uhCAvlBuBq5y3iKwrKJb1ZkdVezTKyDXqp6" style="font-family: Arial; font-size: 11pt; margin-left: 0px; margin-top: 0px; white-space: pre;" width="293" /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Source: <a href="https://dev.to/creativcoder/what-is-a-lsm-tree-3d75 " target="_blank">Creative Coder</a></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">These are two common options. There are other optimization strategies that trade off various constraints.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><b>Reading</b></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Let’s stick with the familiar. A traditional relational database, Oracle, SQL Server, DB2, is row oriented, i.e., stores rows together in files on a drive. Rows offer a good fit for transactional use cases, lots of fine grained, surgical inserts, modifications, etc., and AVRO is a common file format. However, analytical use-cases frequently aggregate one column of data from thousands of rows and thus must read more than they need.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">The heuristic is to store data the way you want to use it. Analytical use cases benefit from column oriented file structures, like Parquet. They read only what they need to and, given their columns, have homogenous data; so they compress to save storage space.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">As always, there is a tradeoff: writes are slower. In fancy language, these structures are great for write once read many workloads. Column database examples, Redshift and HBASE.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><img height="198" src="https://lh6.googleusercontent.com/zkgKQuus6oo9hUaiOQU6CV38FFlRhBcCejV90K-pl73vW6A7f2S0UiJgQmoJjXrwnB3NMc8q8As2vYXBbOQ1rH_Ryx4tvy8JMhry3gPaUNNL4IAxP3ED4UhKP_pcEXR5LC8QAQu6=w441-h198" style="font-family: Arial; font-size: 11pt; margin-left: 0px; margin-top: 0px; white-space: pre;" width="441" /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Source: <a href="http://datacadamia.com" style="font-family: Arial; font-size: 11pt; white-space: pre;" target="_blank">datacadamia.com</a></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Yes, there are hybrids. Cassandra and BigTable can save a set of columns (a family), whose data is frequently used together.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">You may also optimize for data access granularity. If you need to process large grained objects (files) one by one, then an object storage solution like AWSs S3 fits the bill. It uses a key-value pair where the value is the object, e.g. BLOB, Json file, etc. If you need fine grain access to groups of structured data elements, then consider a low-latency, high consistency, key-value/document database solution like AWSs DyanmoDB.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Note the storage types below. It fits this discussion well, but I’ll hand wave it and just provide a link so if you don’t already know, you can learn more about them. </p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><img height="237" src="https://lh4.googleusercontent.com/e2nqineWqHwVED7UBRa3dHyR4Xq8uaGSPqt3t_dN5GtqwVdzLgcc22SJAE0LS5MBJrnFsZUyNxSCRd-Gnnh3N8ahdBN6r4jHACvlpWs3X7ZwAOWbkGnxj_DniPF-5DNW519vOkPz" style="font-family: Arial; font-size: 11pt; margin-left: 0px; margin-top: 0px; white-space: pre;" width="422" /><span style="font-family: Arial; font-size: 11pt; white-space: pre;"> </span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Source: <a href="https://cloudian.com/blog/object-storage-care/" style="font-family: Arial; font-size: 11pt; white-space: pre-wrap;" target="_blank">Cloudain.com</a></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><b>Problem 2: Parallelize work across nodes, i.e., machines.</b></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Since one machine, node, can’t do the work quickly enough, we must parallelize it. We need to abstract away the complexity of dealing with all the different nodes so the developer can focus on the business problem. Map Reduce is one such solution. How it abstracts away the complexity we’ll ignore and simply look at its workflow, which is basic.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Map Reduce does something simple. It collects some data and maps it into a structure, say sales for the week. And then reduces it into something useful, say the min, max, and average sale volumes for the week. The developer writes the logic for the job and the Map Reduce application finds the nodes with the data and runs the logic on each machine in parallel, and then stores it.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><img height="246" src="https://lh4.googleusercontent.com/KvpNtUGkl4_XG-kz1t8aUQYmdU21gTEdRwUnu5_oDkImpAu1KwFkIEHeMVROLdJuGH_WtF2zC4Ci_iZHEYxNBemQ-vELqHyLzkPSPnKQP6FmuY-TVMsLlfGjmt3N5BI0tEhl3AcE" style="font-family: Arial; font-size: 11pt; margin-left: 0px; margin-top: 0px; white-space: pre;" width="329" /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Source: <span id="docs-internal-guid-9b361633-7fff-9737-85f8-de0224ddf25d" style="font-family: Arial; font-size: 11pt; white-space: pre;"><a href="https://livebook.manning.com/book/hadoop-in-practice-second-edition/chapter-1/49" target="_blank">Hadoop In Practice</a></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">While good for simple tasks, the Map Reduce framework causes a developer to create a lot of jobs and knit them together. Imagine quarter-end reporting as a long list of jobs. At Big Data scale, that is difficult to manage. As we’ll see, this is a solved problem.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Big Data Analytics require complex online analytical processing requires inner joins, outer joins, etc. Frameworks like Tez, HIVE abstract away that NoSQL/SQL complexity using algorithms expressed and coordinated via Directed Acyclic Graphs (DAGs). These graphs flow work from left to right and do not loop.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><img height="202" src="https://lh4.googleusercontent.com/AEYAgkgf0OZcbkXtKAjy4a_1w9zbtpIUzm43bGDWaWEQjoQB4sWk_WgrTjbFLzMoSHZI9Pb86UfzJNjIqeanuteJ8nWz_AAvnSiztV_TuBr-dovVb5AM7NwjpUcJIrVaTfSjJ4Lj" style="font-family: Arial; font-size: 11pt; margin-left: 0px; margin-top: 0px; white-space: pre;" width="355" /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Source: <a href="https://medium.com/analytics-vidhya/a-gentle-introduction-to-data-workflows-with-apache-airflow-and-apache-spark-6c2cd9aee573" style="font-family: Arial; font-size: 11pt; white-space: pre;" target="_blank">Medium</a></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">DAGs are expressive and easy to use and have become a de facto developer standard for expressing simple and complex workflows, i.e., they are not just powering low level frameworks. Examples include Spark, Beam, Airflow, Kubeflow, and ML pipelines…</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">ML uses interesting graphs. Feed-forward Neural Networks traverse a graph in one direction, have layers, and evaluate loss at the points where layer nodes meet. Convolutional Neural Networks, and others, use bi-directional traversal, to enable Gradient Descent and Backpropagation.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><img height="161" src="https://lh4.googleusercontent.com/_PqKMPLPP5tcrgNdhwmX5lotjsI-GR-FFobQ7jkHRv7-DZhYXAxEpFVmByun7P0POr2ZDz8jfUNK40z_b_hNp1RDJ3suT3S5x0OH8M21yDChUIIHBPKXzEmAKFQpQ3b0amfN1Kba" style="font-family: Arial; font-size: 11pt; margin-left: 0px; margin-top: 0px; white-space: pre;" width="412" /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Source: <a href="https://brilliant.org/wiki/convolutional-neural-network/" target="_blank">Brilliant.org </a></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Helvetica Neue;"><span style="font-size: 13px;"><b>Problem 3: Efficiently share data and coordinate activity across many nodes.</b></span></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Helvetica Neue;"><span style="font-size: 13px;"><br /></span></span></p><p class="p1" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">Many machines working in parallel mean many logical and physical boundaries to cross, process, user space, system space, network, and storage mediums, etc. Given speed is a goal, boundaries are a toll, they take time to cross.</p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p1" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">Process as much data locally as is practical, I.e. minimize the amount of data passed across the network, processes, or storage boundaries on a node to reduce latency. As mentioned earlier, Map Reduce handles some of this heavy lifting. However, remember it passes data across jobs by persisting it.</p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre;"></span></p><p class="p1" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">As an example, in the Hadoop ecosystem, YARN applications may keep nodes (VMs) active across jobs to keep data in memory for a subsequent workflow step and to avoid VM startup times. The tradeoff is inactive VMs consume memory and may starve others. And there are other examples, Spark uses Resilient Distributed Datasets (RDDs) as a form of shared memory.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><img height="151" src="https://lh3.googleusercontent.com/1l0oGPfai17CSgnVg5I_LMzgzkx-RNK-ibQyveDrB1eRew0XsQlZMGP3TAVPozkeJuRw-2whtGlgbUFzl3_VPmFD1_eGnOkcmJHGc0Rv4xgG9RQhcDzz6hjDjQUxnq3-0SdpqAOV=w491-h151" style="font-family: Arial; font-size: 11pt; margin-left: 0px; margin-top: 0px; white-space: pre;" width="491" /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Source: <a href="https://www.tutorialspoint.com/apache_spark/apache_spark_rdd.htm " style="font-family: Arial; font-size: 11pt; white-space: pre;" target="_blank">tutorialspoint.com</a></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Stating the obvious in distributed systems, avoid workflows that allow hosts to block each other’s tasks, and prefer asynchronous vs synchronous cross node communication. Onto coordination.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Coordination</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Coordination happens at multiple levels. We'll keep it simple. Let’s define “coordinate” to mean managing work across nodes. The next section will address failures.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Continuing to use the Hadoop example, let’s look at the “abstracted complexity” that allows developers to not worry about work distribution. There are three layers:</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">1. YARN (Yet Another Resource Manager) - yep, manages applications, handles some scheduling.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">2. MapReduce - organizes where mapper and reducer work will happen (runs in YARN as of Hadoop 2).</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">3. HDFS - Hadoop Distributed File System - determines where data goes where it is.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><img height="254" src="https://lh3.googleusercontent.com/6DVVeHIB4LlDze0OBDoCFr0jv5dxr8WDTCWHrK_mYOGsdLloYDvO1WGysHBUovUw8n8pxfDLhyhQ97ICX9vuFQAF1fCG3TOWtdJ6S2cSG03U_5QivwuWhcWoKXb4vLB25ymVQXti" style="font-family: Arial; font-size: 11pt; margin-left: 0px; margin-top: 0px; white-space: pre;" width="425" /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Source: <a href="https://livebook.manning.com/book/hadoop-in-practice-second-edition/chapter-1/14 " style="font-family: Arial; font-size: 11pt; white-space: pre;" target="_blank">Hadoop In Practice</a></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">The above diagram highlights a common coordination model, master/slave, or leader/follower, if you prefer. Here, the leader is the authoritative source of what data is “correct” and which node should do what and when. Leader/follower relies on an primary/backup data replication model. A set of changes comes in, let’s say to add a value, and then update it three times. The leader will apply those changes and send the final value to its followers. This can create a single point of failure, so often a backup master who is aware of all the changes. As long as both of them don’t fail at once, all is well. Over two masters? Then there are some overlaps with peer-to-peer. </p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">In centralized distributed computing, leader/follower and peer-to-peer are the main options.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">In peer-to-peer, any node can be a leader or a follower; all nodes have the same capabilities and their function, leader or follower, is context dependent. Here, the data replication model is state machine based. While there is a leader, it shares all incoming changes with the other nodes in its cluster so that anyone of them can become the leader if the current leader fails. <img height="245" src="https://lh3.googleusercontent.com/qSHFvdmHrAwbNpkPJvz9XuvNnunYrI1NLFhAvVWofKq9wnrMmlTdHMAjMWCdoIFNfTHHozrbPzTCznVfSeAGMSIf_N2a4xT-4bx2KzBawcBWHBD_N8SQ95J3Bxy5Jo7dlvikhQTL" style="font-family: Arial; font-size: 11pt; margin-left: 0px; margin-top: 0px; white-space: pre;" width="497" /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Source: <span style="font-family: Arial; font-size: 11pt; white-space: pre;"><a href="https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying" target="_blank">The Log: What every software engineer should know</a></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><span style="border: none; display: inline-block; height: 282px; overflow: hidden; width: 394px;"><img height="282" src="https://lh6.googleusercontent.com/6Wtm4xJWwvAHmVCLRIhmETnm_3PTZXz6UoG5wMHgENTe4qXH0gMHin5gBP1Uk4OhWaD9ObRZnfRjGpdUuufzCdGfasfFdPerYHP1fgKZ19FvQl62cG2soajc0Fv8LbvAdH7xdZlv" style="margin-left: 0px; margin-top: 0px;" width="394" /></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Source: <a href="https://www.tutorialspoint.com/cassandra/cassandra_quick_guide.htm" target="_blank">Tutorialspoint.com</a></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Cassandra uses a peer-to-peer model in a ring topology for replication. Its distribution model is blissfully simple, racks and data-centers. Automated replication across data centers is a popular feature. To be nerdy, very cool.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><img height="345" src="https://lh3.googleusercontent.com/k-2DC0WN23ddnCUpENZHI6VEbizdN5uhPXF7PeHyDNRGtOIH8ndX2mwmIVzqH_lqtc-6jlLYQhkaF2WbULwpPtrsOK4DaW4I25cy0UfVsQKgHinhzXLH7m-G0OMT41d4dBVv6WHF" style="font-family: Arial; font-size: 11pt; margin-left: 0px; margin-top: 0px; white-space: pre;" width="234" /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Source: <a href="https://www.instaclustr.com/blog/cassandra-architecture/ " style="font-family: Arial; font-size: 11pt; white-space: pre;" target="_blank">Intstacluster.com</a></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Your work load should drive your choice. Leader follower is simpler, is easier when strict consistency is required, and with back-up masters is reliable. However, with enough inbound requests, it may bottleneck.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Peer to peer is more complex, is chatty (i.e. requires more internode communication), and prefers eventual consistency. However, this complexity provides dramatic horizontal scalability. For example, a Cassndra deployment ran12,000 commodity servers across many data centers/regions. When deployed properly, you won’t get diminishing returns, it scales linearly.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><b>Problem 4: Reliably process and store data despite one or more node failures.</b></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">At scale we coordinate 100s to 1000s of nodes and serve 1,000,000s of customer. Some of them, hopefully only the nodes, will fail, and sometimes many nodes will fail together. So peers, followers, and leaders all need to share or replicate their data to at least one partner, and frequently 3 to 5 partners. This data might be a customer order or the identity of the current leader on the network when one has failed.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Regardless of the distribution style, node failure in an asynchronous system introduces a hard constraint - it is not possible to be 100% available and provide 100% data consistency. Yes, those are strong words. Read the FLP proof <a href="https://www.the-paper-trail.org/post/2008-08-13-a-brief-tour-of-flp-impossibility/" target="_blank">here</a>. </p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Given three or more copies of every piece of data, that data values change, and that we can’t instantaneously update all nodes, which nodes have the correct value? And what happens when a node fails? Did only it have the correct value? How would you know? Again, we’ll keep it simple here by discussing only a common conceptual approach, the Paxos algorithm.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">The problem is achieving consensus about the correct value of a piece of data at a particular time across multiple nodes when one node can fail. Thankfully, the terminology is straightforward: the nodes must come to a consensus on the correct value, say the value of items in a shopping cart, or perhaps the identity of the current leader.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">The lowly log, append only, is often at the center of achieving consensus during replication in this asynchronous world. It has a lot to do with the order in which things happen. But we’ll leave that for another day.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">The Paxos algorithm in the abstract is simple. Each participant is a node. A node proposes that the other nodes promise to consider a new value for a shared data element. If a quorum of other nodes promise to consider accepting a new value, then the proposer sends out that value. If a quorum of nodes accepts the new value, then the proposer tells all the nodes to commit the value. And how does each node decide? Well, Paxos is a paper unto itself, so read about it <a href="https://martinfowler.com/articles/patterns-of-distributed-systems/paxos.html " target="_blank">here</a>. </p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Cassandra, S3s indexing solution, and many others use Paxos. Other solutions, like Zookeepers atomic messaging, used by Kafka, Hadoop, etc. exist as well. </p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><b>Problem 5: Meet your business needs for data consistency and availability.</b></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">The CAP Theorem is important to understand when talking about distributed systems. It is often misunderstood. CAP stands for Consistency, Availability, and Partitioning. Think of partitioning as a failure, a failure of either a node or a network connection. </p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">The point of the Theorem is that, since we can’t avoid node failure (partitioning) in a distributed system, we must optimize for either consistency or availability. The key word is “optimize.” An AP system can be consistent through failures, but it may take longer than your customer can accept.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">CP System - prioritizes consistency over availability/speed, i.e., I’m willing to wait because I need the most current data</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">AP System - prioritizes availability/speed over consistency, i.e., I’m not willing to wait. Give me what you have now. I’ll deal with it.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">If you must guarantee consistency and Availability AP, then don’t use a distributed system as it may fail, but it can remain consistent.</p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><img height="336" src="https://lh3.googleusercontent.com/LzvlVsv5qsGCdU1ykGgKeFkgz2KhPGDN4k7Hw8pedSlO_IX8eLX8M2cLxi_mgTtoP4r2ipZJ6KtMwt1EII51uFLTW9rINxad684bjfcG2tp9aOVYJADJgzsdq08tAIPy9KBWEmgl" style="font-family: Arial; font-size: 11pt; margin-left: 0px; margin-top: 0px; white-space: pre;" width="406" /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;">Source: <a href="https://medium.com/big-data-processing/cap-theorem-f947a69d0692 " target="_blank">Medium</a></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><br /></p>AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com0tag:blogger.com,1999:blog-5654895526745383189.post-80939068787137300962022-04-07T11:32:00.001-04:002022-07-01T11:35:08.529-04:00Put simply, how does Supervised Learning work?<p></p><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgY5PbO8V-v1PkGHdc60h7C8srltrAFhv10oa0wIUHfLi-NjaBE3n-KJrVA27ub-9sWsrN-BjZqTesZsG4ct1tCm1pVPZnW88WlErlG0XOaNlx8yAFfm5ma-BBb0uKIiddsD59YB6q2xoJShMSRQUVUV7C9qdAwrqY4Po10QQpn1zkmpMN2ALZYHELQqg" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="530" data-original-width="1200" height="141" src="https://blogger.googleusercontent.com/img/a/AVvXsEgY5PbO8V-v1PkGHdc60h7C8srltrAFhv10oa0wIUHfLi-NjaBE3n-KJrVA27ub-9sWsrN-BjZqTesZsG4ct1tCm1pVPZnW88WlErlG0XOaNlx8yAFfm5ma-BBb0uKIiddsD59YB6q2xoJShMSRQUVUV7C9qdAwrqY4Po10QQpn1zkmpMN2ALZYHELQqg" width="320" /></a></div></div><br /><span style="font-family: Arial; font-size: 11pt; white-space: pre-wrap;">In Supervised Learning, a program ingests training data as sets of observations, called features, identified using labels. For example, many emails, each labeled Spam or Not Spam.</span><p></p><span id="docs-internal-guid-c5dd1f09-7fff-a19f-740d-7c52f6fe76e4"><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The program expresses the features mathematically and sends them iteratively into a function. It varies the functions’ governing parameters until getting the desired output. Emails correctly identified as spam or not spam. The program is a machine learning model and the iterative process is called training.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Programs, like people, are not perfect learners. Data scientists and engineers evaluate a model by identifying loss, the number of mistakes made, and variance, how well the model performed across different sets of training data.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Too little loss may cause overfitting where model results are erratic, i.e., have high variance. Too much loss yields more consistent results across data sets, but not accurate enough to be useful.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Data scientists send their programs a lot of data and use a lot of computational resources and time to enable successful machine learning. Sometimes their math enables an accurate model that runs in milliseconds on many data sets, for example, voice recognition on your phone. Sometimes their mathematical approach just fails. Very much a case of “if at first you don’t succeed, try, try again.”</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The math and the technology involved are non-trivial. Today’s data scientists stand on the shoulders of centuries of pragmatic mathematicians. Moore’s law and distributed big data together enable the computing scale required.</span></p><div><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div></span>AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com0tag:blogger.com,1999:blog-5654895526745383189.post-91078502109107204612022-03-11T17:29:00.008-05:002022-03-15T20:01:09.563-04:00Shape Up, Agile Method Summary and Commentary<p> </p><span id="docs-internal-guid-018aeaea-7fff-3355-e340-793f548944e6"><h1 dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhgGwnlb-JQKAtTmt4l9UA979jhvwr11apvaVglGxDSjn-ojL-z_au0z2Ak1d6LGcCDTCjRe2aptvmeCuuTZ7IXCJcYytSJ9CXXsL5XkgM3TVaM3s1M4o2dAXsRAUPFS2g_r2xn0sYJMX8TMt1KIbpDNqHf4Eyy85uMkAa1WKhYPPYukbNgParXGYNLvg" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="706" data-original-width="1102" height="205" src="https://blogger.googleusercontent.com/img/a/AVvXsEhgGwnlb-JQKAtTmt4l9UA979jhvwr11apvaVglGxDSjn-ojL-z_au0z2Ak1d6LGcCDTCjRe2aptvmeCuuTZ7IXCJcYytSJ9CXXsL5XkgM3TVaM3s1M4o2dAXsRAUPFS2g_r2xn0sYJMX8TMt1KIbpDNqHf4Eyy85uMkAa1WKhYPPYukbNgParXGYNLvg" width="320" /></a></div><br /></span></h1><h1 dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">ShapeUp! Shaping, Betting, and Building</span></h1><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The summary starts with the second step in the method, the story flows better this way,</span></p><h2 dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Betting on a six-week release</span></h2><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Key concepts:</span></p><ul style="margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Product and engineering commit to share and mitigate delivery risk (Fix time/effort and vary scope.)</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Cross-functional, autonomous teams align to independent technology components, e.g., services, and have end-to-end accountability for feature design, delivery, and production operations.</span></p></li></ul><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Phase Overview:</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">A 6-week release effort is a “bet” matching an “appetite”</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">An “appetite” is not an estimate. Estimates start with a design and end with a number. Appetites start with a number and end with a design. The appetite is a creative constraint on the design process - to keep it in check, balanced to value.</span></p><ol style="margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Language connotes a business risk worth taking - clear customer value at a reasonable price</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Needs mutual commitment from tech and product to vary scope and approach to win the bet.</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Caps the downside: short enough time to limit the damage if it will cost more than its worth</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Provides pressure: long enough time to get something meaningful done and short enough to feel the date pressure</span></p></li></ol><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Product shapes the next idea as they support the build process of the current idea.</span></p><h2 dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Shaping</span></h2><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Key concepts:</span></p><ul style="margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Shape an idea for a customer outcome to design and build in one release.</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Set boundaries, identify risks, and layout a high-level model, not a design, to be elaborated during the build</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Pitch the idea to place a bet, i.e., to be chosen to attempt delivery</span></p></li></ul><span id="docs-internal-guid-1c6e9280-7fff-5fb2-6d29-7bc8f644d80b"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Phase Overview:</span></span><br /><ol style="margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Start with a </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">raw idea</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> - What problem does it solve? And what outcome gives it customer value? </span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">How will we verify they get it?</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Shape</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> the idea to fit an appetite, </span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">apply design thinking,</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> i.e., it may need to be decomposed, list the constraints.</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Set boundaries</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> - how much is enough?</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Rough out the elements</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> of the idea at a high level, low-fi, but clear on the outcome. Breadth, not depth — explore options. Leave room for designers, e.g., not a UI spec.</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Address risks, and rabbit holes</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> by looking for unintended consequences, unanswered questions, etc. Specify the tricky details.</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Get technical review and </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">determine what is out of bounds.</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Write the pitch:</span></p></li><ol style="margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px;"><li aria-level="2" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">The problem to solve,</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> along with the expected customer outcome and verifier.</span></p></li><li aria-level="2" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Our Appetite</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> - how much time is it worth and what </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">constraints</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> does that imply?</span></p></li><li aria-level="2" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">The core elements of the solution</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> - not the “answer.”</span></p></li><li aria-level="2" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">The Rabbit holes</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> to avoid </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">and risks</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> to deal with.</span></p></li><li aria-level="2" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">No-gos</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> - </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">what should the team exclude</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, things we are choosing not to cover to fit the appetite or make the problem workable.</span></p></li></ol></ol><h2 dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Building</span></h2><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Key concepts:</span></p><ul style="margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Apply design thinking to balance feature design, technical risk, and time to market.</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Organize work in the team by application structures (“scopes”) not people. Scopes are independently buildable and testable and may depend on each other.</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Do the hardest/riskiest thing early</span></p></li></ul><span id="docs-internal-guid-1c6e9280-7fff-5fb2-6d29-7bc8f644d80b"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Phase Overview:</span></span><br /><ol style="margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Product </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">assigns projects, not tasks,</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> and </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">done = deployed</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Hand delivery over to the team</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> to build a feature that gets the outcome given the technology and the time available.</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Discover and </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">map the scopes</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, the independently testable and buildable, end-to-end slices that together make up the feature. Use these scopes, e.g., edit, save, send, to </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">show progress</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">.</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Get one piece done, </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">a small end-to-end slice to gain momentum within a few days</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Start in the middle</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, with the most novel, risky element. If time runs short, simplify or remove nice to haves or should-haves.</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Substance before style - build and verify basic interactions work before focusing on UI styling</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Unexpected tasks and opportunities will appear as you go, so </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">know when to stop</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">.</span></p></li><ol style="margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px;"><li aria-level="2" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Compare completed work to a baseline, e.g. the customer experience now, not a future ideal.</span></p></li><li aria-level="2" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Use the mutual commitment of 6 weeks to an all-or-nothing release as a </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">circuit breaker</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> to limit the scope.</span></p></li></ol></ol><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Commentary</span></p><p style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt; text-align: left;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">Top takeaways:
</span></span></p><ul style="text-align: left;"><li><span style="font-family: Arial;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">Scopes with automated tests speed development and enable the long run product and organizational flexibility that is central to Amazon, Google, Spotify, etc.</span></span></span></li><li><span style="font-family: Arial;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">The “circuit breaker” motivates frequent releases and shared accountability between engineering and product, but may trade-off completeness in the near term.</span></span></span></li><li><span style="font-family: Arial;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">ShapeUp is lightweight and has obvious limits. Say you’ve got 48 people across 8 teams and two years of budget to scale up your software. How do you define and coordinate all of that work? Carefully, I assume.</span></span></span></li><li><span style="font-family: Arial;"><span style="font-family: Arial;">Shape Up heroically assumes autonomous, cross-functional teams. Which I support wholeheartedly, but you may not have. </span></span></li></ul><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">The concepts are excellent and apply outside this lightweight method. I added the italicized content as it felt implied. I’m guessing most who use it would also sprinkle in some scrum. This approach begs for XP practices. Scopes are natural outcomes of TDD and BDD.</span><p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The summary leaves a lot out, e.g., large project how-to guidance. Some of it seemed silly, e.g., visualizing status as scopes rolling up and down hills. But, the content is free; you find it here, and I’m not complaining. The scopes concept is central to software development. In fact, outside of the hill thing, there is little to dislike.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">ShapeUp is lightweight and has obvious limits. Say you’ve got 48 people across 8 teams and two years of budget to scale up your software. How do you define and coordinate all of that work? Carefully, I assume. It heroically assumes autonomous, cross-functional teams. Which I support wholeheartedly, but you may not have.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 12pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">A small consultancy doing web development projects for clients created the method, and it fits that like a glove. It can be a great fit for small tech startups until they scale past two or three teams. After that, keep the concepts and solve the next set of problems. Lots of good toolsets out there, LeSS, Scrum, etc.</span></p><br /></span>AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com0tag:blogger.com,1999:blog-5654895526745383189.post-78924614871968899922022-03-04T10:11:00.008-05:002022-07-01T10:00:23.352-04:00My most popular content...<p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEi86d-2lb1c8W3c7KstDqyUgvwlSbSYfse0CBjo99jrZCdl_H6U8mmG6KrSYGTYC-X58zyUzGlKRsG2XFiAwRZuslzaTnDqfGkrJL2oAoHg6qHGQw5UowWDoU2oqhjmiX9SWDjcWVyRDBI1ND6cwQ6tKqeI8JrLfaEkNBk8OEMiTyrZtLPHswQc5-roZg" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="482" data-original-width="724" height="213" src="https://blogger.googleusercontent.com/img/a/AVvXsEi86d-2lb1c8W3c7KstDqyUgvwlSbSYfse0CBjo99jrZCdl_H6U8mmG6KrSYGTYC-X58zyUzGlKRsG2XFiAwRZuslzaTnDqfGkrJL2oAoHg6qHGQw5UowWDoU2oqhjmiX9SWDjcWVyRDBI1ND6cwQ6tKqeI8JrLfaEkNBk8OEMiTyrZtLPHswQc5-roZg" width="320" /></a></div><p>Prior to 2022, google analytics shows my most popular content is the book summaries. So, here's a list with direct links. </p><p></p><ol style="text-align: left;"><li><a href="https://agilejazz.blogspot.com/p/the-devops-handbook-summary-1-of-4.html">The DevOps Handbook</a> - a four-part book summary</li><li><a href="https://agilejazz.blogspot.com/p/lean-start-up-excerpt-based-summary.html">Lean Start-Up</a> - a three-part book summary</li><li><a href="https://agilejazz.blogspot.com/p/kanban-book-summary.html">Kanban</a> - a seven-part book summary</li></ol><div>If you like this type of content then you are sure to enjoy these as well:</div><div><ol style="text-align: left;"><li><a href="https://www.leanessays.com/2019/" target="_blank">The Nature of Software</a> - Mary Poppendieck (30-minute read)</li><li><a href="https://www.agilefluency.org/" target="_blank">Agile Fluency</a> - The Agile Fluency Project</li><li><a href="https://less.works/" target="_blank">Large Scale Scrum</a> - the LeSS framework by Craig Larman & Bas Vodde</li></ol></div><div>If you are looking for an afternoon read, Doc Norton's book <a href="https://leanpub.com/escapevelocity" target="_blank">Escape Velocity</a> is excellent. A short book but packed with good software delivery practice wisdom. </div><div><br /></div><div>Have fun!</div><p></p><p></p>AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com0tag:blogger.com,1999:blog-5654895526745383189.post-65962566715903932052017-01-21T17:34:00.018-05:002022-01-31T12:50:50.969-05:00Has Agile Lost It's Way?<div style="text-align: center;">
<br /></div>
<div style="text-align: left;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaeL5upfaoWdD4qfXAvbB1WFW3XJVja85GlHEdzMaZP058W1qWsSmkn2pL9LDauhcCLhrUXSp4x1Sy9LYCZlgIaL_aUWDignIpGe-w7hgeiw6WxGkwpzLAq2fV10RFR-uellGn0J0Ercxw/s1600/lostway.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="170" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaeL5upfaoWdD4qfXAvbB1WFW3XJVja85GlHEdzMaZP058W1qWsSmkn2pL9LDauhcCLhrUXSp4x1Sy9LYCZlgIaL_aUWDignIpGe-w7hgeiw6WxGkwpzLAq2fV10RFR-uellGn0J0Ercxw/s400/lostway.jpg" width="400" /></a></div>
<div style="text-align: center;">
<br /></div>
When I got started with Agile in 2007 it was new. Most of us had been delivering software for a while and while we loved developing solutions we hated the often tyrannical circumstances we endured doing it. We knew what worked: lots of unit testing, iterative/incremental development, lots of collaboration, transparency, etc. But until Agile (for me, XP, and Scrum), we didn't know how to pull it all together. What we found, before Daniel Pink gave simple ways to say it, was that when we had the opportunity to 'do it right,' we were on a path to mastery, autonomy, and purpose in our daily work.</div>
<div style="text-align: left;">
<br />
Agilists associate the birth of "Agile" with <a href="https://www.martinfowler.com/articles/agileStory.html">The Snowbird Conference</a> in 2001 and <a href="http://agilemanifesto.org/">The Agile Manifesto</a>. If we focus on the unmet needs of businesses and governments relying on software then <a href="https://techbeacon.com/agility-beyond-history%E2%80%94-legacy%E2%80%94-agile-development">it goes farther back still</a>. Let's just say Agile's about 16, a teenager. It shows.<br />
<br />
Agile has become a big business promising <b>homogenized</b> faster, better, and cheaper delivery to the masses. I would guess that 70% or more of teams who call themselves Agile aren't. At least by my definition but, I'm a tough grader who expects the use of XP engineering practices.<br />
<br />
In 2012 Thoughtworks' published an <a href="https://martinfowler.com/articles/agileFluency.html">Agile Fluency Model</a> to describe levels of Agility from One Star Teams focused on delivering business value to Four Star Teams optimizing entire eco-systems. Their data suggested that 15% of teams were doing something, but it wasn't Agile. 45% of teams, while focusing on business value, weren't meaningfully improving software quality. As a result, due to Scrum or Scrum-like processes, these teams are only marginally more productive.<br />
<br />
<div style="text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3ynbJROX8PYtQv7Nw_mtBm70tj3ot-q1NMIJAF8OlpCAuvxry_sft_onXzAQtRU3_vw0FU-eG7VO7XZOic-neU14PFTYrwJBr7WarPk-HQemshfWMbnvBdvpKbGecUrIjTaWH_m9bNjl-/s1600/ButWaitTheresMore.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3ynbJROX8PYtQv7Nw_mtBm70tj3ot-q1NMIJAF8OlpCAuvxry_sft_onXzAQtRU3_vw0FU-eG7VO7XZOic-neU14PFTYrwJBr7WarPk-HQemshfWMbnvBdvpKbGecUrIjTaWH_m9bNjl-/s200/ButWaitTheresMore.png" width="198" /></a></div>
<br />
As a result of this homogenization, the tyrannical circumstances Agile was to dispel are back for too many. As the market grew, "consultants" with no more than a scrum class and a half-read, frequently derivative book popped up to feed on the Agile sales frenzy, the essence started getting lost.<br />
<br />
Turns out that greater transparency can also lead to micro-management and ever more unrealistic expectations on a team. You get worse, not better code. You get less not more shared understanding. Even when you've had some success by focussing solely on visible business value, you may hear, "Wow, we've gotten faster since leaving waterfall. Bet you can go twice as fast! Indeed, you must! Find more process improvements!"<br />
<br />
If this is sounding familiar to you I'm sorry. Here's what I suggest you might reference to find the essence:<br />
<ol>
<li>Read the original sources:</li>
<ul>
<li><a href="https://www.amazon.com/Extreme-Programming-Explained-Embrace-Change/dp/0321278658">Extreme Programming Explained: Embrace Change</a></li>
<li><a href="https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&sqi=2&ved=0ahUKEwiessfS7NPRAhUJ7YMKHdPADtkQFggaMAA&url=http%3A%2F%2Fwww.scrumguides.org%2Fdocs%2Fscrumguide%2Fv2016%2F2016-Scrum-Guide-US.pdf&usg=AFQjCNFFR6gkteQ492jBZbtqmJzNvgDcbg&sig2=sbi9Shy5pL9ydgszXIoYPQ&bvm=bv.144224172,d.amc">The Scrum Guide</a></li>
</ul>
<li>If you are a developer or architect then also check out:</li>
<ul>
<li><a href="https://www.amazon.com/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882">Clean Code: A Handbook of Agile Software Craftsmanship</a></li>
<li><a href="https://cleancoders.com/">Clean Coders</a>: watch as many videos as you can afford</li>
</ul>
<li>If you are an Architect add these to the above:</li>
<ul>
<li>Read <a href="https://www.amazon.com/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882">Clean Code</a></li>
<li>Watch the Architecture, Tech Debt, and TDD videos at <span id="goog_1413407644"></span><a href="https://cleancoders.com/">Clean Coder<span id="goog_1413407645"></span>s</a></li>
<li>If you haven't written a line of code in 10 years, and think this is all craziness then spend the next month coding, full-time in your enterprise applications then read and watch the videos again.</li>
</ul>
</ol>
<div style="text-align: left;">
Once you are done, happy New Years' 2006! Please don't stop now. Next up:</div>
<div style="text-align: left;">
<ol>
<li><a href="https://www.youtube.com/watch?v=u6XAPnuFjJc">RSA ANIMATE: Drive: The surprising truth about what motivates us</a></li>
<li><a href="https://www.amazon.com/Kanban-Successful-Evolutionary-Technology-Business/dp/0984521402/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=1485024314&sr=1-1">Kanban: Successful Evolutionary Change for Your Technology Business</a></li>
<li><a href="https://www.amazon.com/Large-Scale-Scrum-More-Addison-Wesley-Signature-ebook/dp/B01JP91OR4/ref=sr_1_1?s=books&ie=UTF8&qid=1485024181&sr=1-1&keywords=bas+vodde">The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses</a></li>
<li><a href="https://www.amazon.com/Large-Scale-Scrum-More-Addison-Wesley-Signature-ebook/dp/B01JP91OR4/ref=sr_1_1?s=books&ie=UTF8&qid=1485024181&sr=1-1&keywords=bas+vodde">Large-Scale Scrum: More with LeSS</a></li>
<li><a href="http://itrevolution.com/devops-handbook">The DevOps Handbook</a></li>
<li><a href="https://www.amazon.com/Badass-Making-Awesome-Kathy-Sierra/dp/1491919019/ref=sr_1_1_twi_pap_2?s=books&ie=UTF8&qid=1485026115&sr=1-1&keywords=badass+making+users+awesome">BADASS: Making Users Awesome</a></li>
</ol>
Ok, happy New Years' 2017! We're not done. So much has happened since Agile was born. Not all of it captured above. Keep in mind that you have to know the rules before you can successfully break them. You should also check into <a href="http://modernagile.org/">Modern Agile</a>. </div>
<br />
<div style="text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEislQinTtWK9R2dE4qnq-MpgrtLUL2PMHx5jvMkTXyPc3Xj3d95ld5bA9cto8dk9ojeTivOMNonSfJVP8Za_Tt8-NcDeNlmBtZLn0s8oLYEDjESK6JACygudDdXrX78iNmIHIBw1bxL-Hob/s1600/Screen+Shot+2016-09-14+at+9.21.29+AM.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="293" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEislQinTtWK9R2dE4qnq-MpgrtLUL2PMHx5jvMkTXyPc3Xj3d95ld5bA9cto8dk9ojeTivOMNonSfJVP8Za_Tt8-NcDeNlmBtZLn0s8oLYEDjESK6JACygudDdXrX78iNmIHIBw1bxL-Hob/s320/Screen+Shot+2016-09-14+at+9.21.29+AM.png" width="320" /></a></div>
<br />
<br /></div>
AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com10tag:blogger.com,1999:blog-5654895526745383189.post-68818855339470383322017-01-13T11:11:00.003-05:002017-01-14T10:31:53.942-05:00Finally Finished the Kanban Book Summary!Yes! Finally finished the seven-part summary of "Kanban: <a href="https://www.amazon.com/Kanban-Successful-Evolutionary-Technology-Business/dp/0984521402">Successful Evolutionary Change for Your Technology Business</a> by <a href="http://anderson.leankanban.com/">David Andersen</a>. Part one starts <a href="http://agilejazz.blogspot.com/p/kanban-book-summary.html">here</a>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYb3TaYWGZm3rAOItNBBFHdlPmebxMRufzZpeA6CEZXz54A7jHlBdf_SgXfEx6EerIguhwsfjr1qV2wOj5KTmIet6O6_gB8gaNcHTCtL2DFpwvKtXkN5MLh1RU59bkcOaOKxA87L__u4Bc/s1600/Screen+Shot+2017-01-13+at+11.12.36+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYb3TaYWGZm3rAOItNBBFHdlPmebxMRufzZpeA6CEZXz54A7jHlBdf_SgXfEx6EerIguhwsfjr1qV2wOj5KTmIet6O6_gB8gaNcHTCtL2DFpwvKtXkN5MLh1RU59bkcOaOKxA87L__u4Bc/s320/Screen+Shot+2017-01-13+at+11.12.36+AM.png" width="259" /></a></div>
<br />
<br />
Of all the summaries so far, this is one of my favorites. The summaries are written as study guides or review materials.<br />
<br />
The Kanban book fills in the blanks regarding why Scrum and XP work the way they do <i>and</i> how to improve their processes in a way that minimizes resistance to change and maximizes value delivered.<br />
<br />
Kanban is not a methodology for software delivery. It is a change management system for improving the implementation of any delivery framework or process. It is reasonable to say that using the system to improve a waterfall process would lead to Scrum/XP/Lean Start Up like processes. This may explain why some think Kanban is a delivery methodology - applying its principles will suggest process improvements that likely converge towards the processes that are taken to define today's "Agile."<br />
<br />
But that too is perhaps misleading. Agile is a mindset growing from of a set of principles and values we apply to improving our ability to deliver value through software delivery. Is is not a process or a framework - despite industry marketing machine claims to the contrary. It uses processes and frameworks and it is our job to improve them. If you agree then you'll enjoy getting involved with <a href="http://modernagile.org/">Modern Agile</a>.<br />
<br />
David Andersen's application of Lean principles to software delivery and their synthesis with the Agile mindset makes Kanban a valuable tool. Here are the summaries:<br />
<br />
<ul>
<li><a href="http://agilejazz.blogspot.com/p/kanban-book-summary.html">Kanban Summary 1 of 7: Intro, Benefits, Kaizen</a></li>
<li><a href="http://agilejazz.blogspot.com/p/kanban-summary.html">Kanban Summary 2 of 7: Value Stream and Basic System Cadences</a></li>
<li><a href="http://agilejazz.blogspot.com/p/this-is-summary-of-book-kanban.html">Kanban Summary 3 of 7: WIP Limits and Classes of Service</a></li>
<li><a href="http://agilejazz.blogspot.com/p/kanban-summary-metrics.html">Kanban Summary 4 of 7: Metrics, Projects, and Operational Reviews</a></li>
<li><a href="http://agilejazz.blogspot.com/p/this-is-summary-of-book-kanban_19.html">Kanban Summary 5 of 7: Goals, 12 Steps, and Striking a New Bargain</a></li>
<li><a href="http://agilejazz.blogspot.com/p/kanban-summary-6-of-n.html">Kanban Summary 6 of 7: Improve - Bottlenecks and Waste</a></li>
<li><a href="http://agilejazz.blogspot.com/p/book-summary-of-kanban-successful.html">Kanban Summary 7 of 7: Improve - Reduce Variability, Resolve Issues</a></li>
</ul>
<br />
<br />AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com4tag:blogger.com,1999:blog-5654895526745383189.post-56651778792308045442017-01-01T19:08:00.000-05:002022-01-25T11:13:22.695-05:00Old Originals (1978-1993)<br />
<div>
In no particular order...</div>
<ol>
<li><a href="https://drive.google.com/open?id=1YmwI6e_dB3MQR44VWOrGUwCAtyS2QmzR">Ande</a></li>
<li><a href="https://drive.google.com/open?id=1r_bae658e5_ToAS1ugNuQhpeuQGBms5n">As I Fell</a></li>
<li><a href="https://drive.google.com/open?id=1oUiB9ojvRkTYxqjxepzoAUjJIuK7RTfp">Feel Male Persuasion</a></li>
<li><a href="https://drive.google.com/open?id=181pGh9AMOzfSih_DoBbCYb1z0KaZXOP0">Fool for your love</a></li>
<li><a href="https://drive.google.com/open?id=128ZB2-2ARwWkWfm_Az6y6JEHnh2CM5aq">London</a></li>
<li><a href="https://drive.google.com/open?id=1J_Hef5sj3nlWbJPWIGVoOjoKK0ieBJSB">Road Sketches</a></li>
<li><a href="https://drive.google.com/open?id=1b6CaqH3oseET9nS62ueE3uwpRCDo97H3">Simple</a></li>
<li><a href="https://drive.google.com/open?id=12DE3aqfGkcZ0KAKh44X4VRuP6faSyo2F">Teas'n your persuasive</a></li>
</ol>
<br />
<br />AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com13tag:blogger.com,1999:blog-5654895526745383189.post-87468753271770463992016-12-08T12:38:00.001-05:002016-12-23T09:36:37.782-05:00Big Data Technology Impacts of NoSQL, Distributed Data Stores...The last 30 years have brought dramatic change to the world and technology. Moore's law and the internet's relentless march connect more devices than there are people on the planet. Cisco estimates 50 billion connected devices by 2020. To many cloud computing, big data and NoSQL tools sit hidden behind Facebook, Google and ubiquitous smart phone apps . Even those in IT may sometimes struggle to synthesize these innovations.<br />
<br />
In this post we'll take a brief look at the implications of some of these changes on traditional business intelligence and analytic capabilities. Think of this as a quick survey of the landscape and not a deep dive into its many facets. Let's start with a seemingly esoteric NoSQL capability: graph processing and databases.<br />
<br />
<h4>
Graph Processing and Databases</h4>
We benefit from graph technology every day. Let's look at a visualization and some use cases.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguAfL6SrBL4omnNAKdPeu6Oyzp0vcr-c9Mp4N2_8Ac6gm5XK25WYNxGWPuMRPMeiTumCEPuzhdp_S4d6Xuj8ym_ABMIdfGJ08eKgDlJcz_FXiW9xKMKSSRjpZBJL8qnyizBJzGcO6DCy6E/s1600/Screen+Shot+2016-12-06+at+1.53.58+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="166" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguAfL6SrBL4omnNAKdPeu6Oyzp0vcr-c9Mp4N2_8Ac6gm5XK25WYNxGWPuMRPMeiTumCEPuzhdp_S4d6Xuj8ym_ABMIdfGJ08eKgDlJcz_FXiW9xKMKSSRjpZBJL8qnyizBJzGcO6DCy6E/s200/Screen+Shot+2016-12-06+at+1.53.58+PM.png" width="200" /></a></div>
<br />
Real time recommendations on retail web sites, the sometimes creepy "friend recommendations" on Facebook. Both echo chamber and gold mine: social network analytics for marketing and advertising. GPS applications like Waze or google maps. A little less obvious are use cases like network and operations root cause analysis, contextualizing real time streams of event and other data from the internet of things' sensors. Finally, saving money and sometimes lives via fraud detection, cybersecurity, and medical research. <br />
<br />
When the relationships of things to each other are as important as the things themselves - graphs are attractive. Beyond the ubiquity of this hidden technology notice that unlike traditional BI and analytics for some key use cases, e.g. recommendations, identity and access management, it can be real time.<br />
<br />
Before we move on to look at NoSQL more generally, it's important to recognize a few things about graph processing and graph databases. Graph databases often compete as OLTP tools e.g. N<a href="https://neo4j.com/">eo4j</a>. Large scale graph processing is more an OLAP capability e.g. <a href="http://giraph.apache.org/">Giraph</a>, <a href="https://blog.acolyer.org/2015/05/26/pregel-a-system-for-large-scale-graph-processing/">Pregel</a>, so the graph's data may sit in another NoSQL distributed data store. Having said that, the intuitive nature of the graph database lends itself to OLAP. <br />
<br />
Depending on the scale and type of the analysis graph queries using languages like cypher or gremlin are often dramatically simpler to write than similar queries in SQL. Also, queries across many layers of related things perform orders of magnitude faster in a graph database than in a relational database. <br />
<br />
Before we go any farther we'll need to have at least a passing understand of how NoSQL enables big data.<br />
<br />
<h4>
NoSQL - A Quick Overview</h4>
<div>
What's happened to the "relational" world? Data warehouse data marts and multi-dimensional cubes for OLAP, along with SAS, once ruled the BI and data analysis landscape. Yet open sourced projects like <a href="https://en.wikipedia.org/wiki/R_(programming_language)">R</a>, <a href="https://en.wikipedia.org/wiki/Apache_Hadoop">Hadoop</a>, <a href="https://en.wikipedia.org/wiki/Apache_Cassandra">Cassandra</a>, <a href="https://en.wikipedia.org/wiki/Apache_Spark">Spark</a> are in the forefront of the now "big" data world. Even SQL database OLTP dominance may be challenged by the modern graph databases like Neo4j's ACID support, ease of use, and lightening speed for some use cases. </div>
<div>
<br /></div>
<div>
NoSQL - could anyone have picked a worse or more misleading name? Every major cloud provider offers SQL based API's to many of their "noSQL" data sources. SQL tends to connote relational data stores and tables. Yet, graph databases are intrinsically relational and the open sourced versions of google's distributed, non-relational data store (now categorized as Column based noSQL) is called <b>BigTable</b>/MapReduce. Oh well, at least Column based noSQL is not really tabular. <br />
<br />
NoSQL type characteristics overlap suggesting innovation will continue. My advise is to think "not only SQL" when you think of NoSQL.</div>
<div>
<br /></div>
<div>
Wikipedia's current taxonomy of relevant types of <a href="https://en.wikipedia.org/wiki/NoSQL">NoSQL</a> along with example implementations.</div>
<div>
<ul>
<li>Column: Cassandra, <a href="https://en.wikipedia.org/wiki/Apache_HBase">HBase</a></li>
<ul>
<li>This type is often part of a distributed data store which has many benefits beyond the scope of this conversation. For now we'll just say it allows reliable, fast querying agains huge amounts of data across many computers, or nodes, at once</li>
<li>Map Reduce often handles querying where "map" sorts and filters and "reduce" summarizes. Other common query tools include <a href="https://en.wikipedia.org/wiki/Apache_Hive">Hive</a> and <a href="https://en.wikipedia.org/wiki/Pig_(programming_tool)">Pig</a></li>
<li>A column of a distributed data store is its lowest level object. It is a tuple (a key-value pair) consisting of three elements: A unique name, a value (or set of values), and a time stamp to determine if the content is valid or stale.</li>
<li>Example columns: </li>
<ul>
<li>street: {name: "street", value: "1234 x street", timestamp: 123456789},</li>
<li>city: {name: "city", value: "san francisco", timestamp: 123456789}</li>
</ul>
<li>This data often resides on a Hadoop filesystem and may be performance optimized via Spark</li>
</ul>
<li>Graph: Neo4J, Apache Giraph, MarkLogic</li>
<ul>
<li>Graph databases uses nodes which represent entities like person, edges which represent relationships, and Properties which can be associated with both nodes and edges</li>
<li><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTbezpSvElI4KlU9Up_j44rUEEjGAcnUYSdlHe1O1Uch0xBlccGNGaaQP8SZei7yuuL8rHNns0zNnea-_FoS08fQGBDx_UY37QDTa__qLg4LXlqxzPej1piC2mZkTUByNUbypH17rFsj8z/s1600/Screen+Shot+2016-12-06+at+4.24.43+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" height="225" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTbezpSvElI4KlU9Up_j44rUEEjGAcnUYSdlHe1O1Uch0xBlccGNGaaQP8SZei7yuuL8rHNns0zNnea-_FoS08fQGBDx_UY37QDTa__qLg4LXlqxzPej1piC2mZkTUByNUbypH17rFsj8z/s320/Screen+Shot+2016-12-06+at+4.24.43+PM.png" width="320" /></a></li>
<li>Working with graph databases is generally intuitive as the storage model directly reflects natural language rather than being burdened with the physical implementation details of most other data stores e.g. relational or column</li>
<li>Relational databases use keys to relate entities to one another leading to "joining" one to many tables. Graph databases use pointers to relate one entity to another and may capture properties about the relationship. The deeper the levels of relationships the more this database differentiates itself from relational stores.</li>
<li>Compared with relational databases, graph databases are often faster for associative i.e. highly related, data sets and map more directly to the structure of object-oriented applications. They can scale more naturally to large data sets as they do not typically require expensive join operations. As they depend less on a rigid schema, they are more suitable to manage ad hoc and changing data with evolving schemas. </li>
<li>Conversely, relational databases are typically faster at performing the same operation on large numbers of data elements.</li>
</ul>
<li>Multi-model: MarkLogic, OrientDB</li>
<ul>
<li>Supports multiple data models against a single, integrated backend. Document, graph, relational, and key-value models are examples of data models that may be supported by a multi-model database.</li>
<li>An article by Martin Fowler suggests this type will become more prevalent over time <a href="http://martinfowler.com/bliki/PolyglotPersistence.html">Polyglot Persistenc</a>e</li>
</ul>
<li>Document: MarkLogic, MongoDB</li>
<ul>
<li>A document-oriented database, or document store, is designed for storing, retrieving, and managing document-oriented information</li>
<li>Graph databases are similar, but add another layer, the relationship, which allows them to link documents for rapid traversal.</li>
</ul>
<li>Key-Value: <a href="https://en.wikipedia.org/wiki/Redis">Redis</a>, Oracle NoSQL</li>
<ul>
<li>Manages data in associative arrays, a data structure more commonly known today as a dictionary or hash. </li>
<li>Dictionaries contain a collection of objects, or records, which in turn have many different fields within them, each containing data. These records are stored and retrieved using a key that uniquely identifies the record, and is used to quickly find the data within the database.</li>
<li>Very easy for developers to use e.g. to persist objects. Computationally powerful for some use cases. Some graph databases underlying implementations are key-value store.</li>
</ul>
</ul>
<div>
<br /></div>
<h4>
Dealing with the Big in Big Data</h4>
</div>
<div>
<b>A simple example:</b></div>
<div>
Let's take a simple comparison: doing analysis of invoices versus sensor data from the internet of things. Traditionally for invoices you might extract the invoice data from your transactional data store, transform it into dimensions and facts, load it into your data mart to run reports or do sophisticated analysis by loading it into a cube. For gigabytes of data this works great (if requiring a lot of money to set up - more on that in a minute). </div>
<div>
<br /></div>
<div>
Sensor data sets are often much, much larger, think terabytes. Yet, we can simplify and say the data has many dimensions (e.g. time, sensor type) and sensor readings (facts) to assess. Using ETL to create a data mart is impractical. Enter "big data" with its distributed data stores and analytic capabilities.</div>
<div>
<br /></div>
<div>
Let's take a more accessible, if over simplified, example than sensor data. Perhaps Macy's wants to do analysis of the last 50 years of sales invoices. Rather than put all of that data into a data mart they might leverage the distributed computing power of Hadoop and MapReduce. They could create a text file for each invoice and then distribute the invoices across many data stores in the cloud. Next they'd leverage MapReduce to do their summarizations across many computers in parallel.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8t-MB6A2SXoyDuPkOzNn0SEZOXF543xdryL0TWpF8ufuBCPbhq8a_wqFcLpMvMpU8jLgSUVc7vtH1u1XoxX7i6e9eQFG7sD7I_hAdpG_1bQAPAraMB0Lcy_WZNnxIKnGcRp0rPAgtE-VE/s1600/Screen+Shot+2016-12-08+at+10.45.53+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="293" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8t-MB6A2SXoyDuPkOzNn0SEZOXF543xdryL0TWpF8ufuBCPbhq8a_wqFcLpMvMpU8jLgSUVc7vtH1u1XoxX7i6e9eQFG7sD7I_hAdpG_1bQAPAraMB0Lcy_WZNnxIKnGcRp0rPAgtE-VE/s320/Screen+Shot+2016-12-08+at+10.45.53+AM.png" width="320" /></a></div>
<div>
Turns out this processing can be done very quickly. Programming is involved but a no where near the cost of using a more traditional approach. The aggregations etc. that this processing does can then be fed into a tool like SAS (or R) to do more in-depth, ad-hoc analytics. </div>
<div>
<br /></div>
<div>
<b>Getting more granular:</b></div>
<div>
The term "text file" above could use some expansion. These could be realized as NoSQL columns, hive "tables" etc. Hive is a great example of why we might choose "not only SQL" as the expansion of the term NoSQL. It exposes a SQL API to interact with virtual tables. </div>
<div>
<br /></div>
<div>
As the complexity of the problem solved grows so does its implementation details. For example, designing the structure of an HBASE database is non-trivial. </div>
<div>
<br /></div>
<div>
In the simple example above we say the outputs of MapReduce can then be input to a tool like SAS for ad-hoc querying. Apache SPARK enables Hive to present users complex, OLAP style ad-hoc query capabilities agains huge distributed data stores. </div>
<div>
<br /></div>
<div>
Spark creates a flexible abstraction called the "resilient distributed dataset" that aggregates data across many computers in a cluster. This aggregation overcomes MapReduce's limitation of requiring a linear data flow where you map a function across data and then reduce the results onto disk. It overcomes this limit by creating shared memory across the computers in the cluster enabling iterative passes over the data. Put more simply, it brings OLAP (and machine learning capabilities) to cloud tools including Hadoop's HDFS, Cassandra, Amazon S3, Open Stack's Swift...</div>
<div>
<br /></div>
<div>
However, Spark also necessitates additional complexity such as the use of cluster managers. It has its own native manager as well as provides support for <a href="https://en.wikipedia.org/wiki/Apache_Mesos">Apache Mesos</a> and Hadoop YARN. Once again we see google innovation at work. Mesos conceptually descends from google's omega scheduler. It has used Omega to manage its services at scale. We'll stop here as cluster managers are a topic onto themselves.</div>
<div>
<br /></div>
<h4>
Finishing up</h4>
<div>
This post summarizes a huge space in a few pages. I hope you found it useful. I've tried to describe a very complicated space in a simple manner. If I've misrepresented rather than simplified, please comment so I might update this post.</div>
<div>
<br /></div>
<div>
So we started big lets end big. Facebook's network of friends has caused it to be a pioneer in the graph processing. They have scaled graph processing up to handle a trillion edges ie. relations! Read about that <a href="https://www.facebook.com/notes/facebook-engineering/scaling-apache-giraph-to-a-trillion-edges/10151617006153920/">here</a>. Here's a visual of their conceptual architecture.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj05KDS9aHE9JYRgfsmWjL6eLbmNIAX-pkS8CQZfny9TS7awOz4a7URI0AK7izoSml9UrkIgaIXuvMszpMF_IwTWrpKkRBDo0syNUTuHsohultvQEBk2G0Nv3eUmYhIYQLLejnqeLpjupVA/s1600/Screen+Shot+2016-12-08+at+12.37.11+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="150" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj05KDS9aHE9JYRgfsmWjL6eLbmNIAX-pkS8CQZfny9TS7awOz4a7URI0AK7izoSml9UrkIgaIXuvMszpMF_IwTWrpKkRBDo0syNUTuHsohultvQEBk2G0Nv3eUmYhIYQLLejnqeLpjupVA/s320/Screen+Shot+2016-12-08+at+12.37.11+PM.png" width="320" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com2tag:blogger.com,1999:blog-5654895526745383189.post-16258183426048615422016-12-05T09:39:00.004-05:002016-12-08T06:42:02.783-05:00Big Data Business Intelligence and Visualization 2016Early in my career at Accenture I spent a lot of time consulting in Enterprise Information Management or more specifically: Content Management and Publishing (IA), Data Strategy and Governance, and bit in BI. After spending some years in mostly Agile, OLTP, and SOA, job opportunities present themselves that have me looking into the space again. This post is the first of a few on developments here. This one looks at Gartner's 2016 vendor POV and some self-service Data Visualization vendors.<br />
<br />
With the advent of big data technologies like Hadoop, Map Reduce, Spark, Graph DB's, <a href="https://neo4j.com/developer/graph-db-vs-rdbms/">Neo4J</a>, <a href="https://blog.acolyer.org/2015/05/26/pregel-a-system-for-large-scale-graph-processing/">Pregel</a>... ....<a href="https://en.wikipedia.org/wiki/Data_visualization#References">data visualization</a>, <a href="https://en.wikipedia.org/wiki/Multidimensional_network#Software">multi-dimensional networks</a>, IOT and cloud this space is ripe with opportunity. Gartner's <a href="https://www.gartner.com/doc/reprints?id=1-2XXET8P&ct=160204">2016 Magic Quadrant for Business Intelligence and Analytics Platforms</a> illustrates some of this. They have fundamentally retooled how they look at the space.<br />
<br />
<h4>
Gartner's view is generally: </h4>
<br />
<ol>
<li>The market is moving from centralized IT BI capability delivery to decentralized self-service business capabilities often baked into business processes. Business self-service data preparation and analysis is a good example. </li>
<li>From a technology perspective: "By 2018, smart, governed, Hadoop-based, search-based and visual-based data discovery will converge in a single form of next-generation data discovery that will include self-service data preparation and natural-language generation."</li>
<li>The vendor market is fragmented. Trying to rely on one big vendor will likely not meet all of your needs. In the near term businesses should assess their needs and find the products that match them.</li>
</ol>
<h4>
Gartner's five use cases reflect this view:</h4>
<br />
<div>
<ol>
<li>Agile Centralized BI Provisioning</li>
<li>Decentralized Analytics</li>
<li>Governed Data Discovery</li>
<li>Embedded BI</li>
<li>Extranet Deployment</li>
</ol>
<div>
Before we look at the Magic Quadrant let's look at overall vendor group's performance against Gartner's 2016 "Critical Capabilities for Business Intelligence and Analytics Platforms" publication. It is interesting that the rankings, using a 1-5 scale, on average are not that good. I am not sure if this is a common occurrence in their assessment methodology. The rankings may simply reflect market conditions.</div>
</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgq70djw0mNIsvfapbcgds_yYP_2UMTZZkt5xKxABPD1NyyKDPTmiRVW9HLq3cvTpTHmg7shHKStiU1Pgldrazpso3Mbawa3NzeMbi6OggPYAZolYPhZFsTztc_ue-BFgyCtyHGFbluaNQL/s1600/Screen+Shot+2016-12-05+at+8.49.59+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="331" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgq70djw0mNIsvfapbcgds_yYP_2UMTZZkt5xKxABPD1NyyKDPTmiRVW9HLq3cvTpTHmg7shHKStiU1Pgldrazpso3Mbawa3NzeMbi6OggPYAZolYPhZFsTztc_ue-BFgyCtyHGFbluaNQL/s400/Screen+Shot+2016-12-05+at+8.49.59+AM.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
The group's only "excellent" capability average comes in "BI Platform Administration." Looks like cloud may be helping to level the playing field a bit for new entrants. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The overall vendor results are similarly mixed:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgS6EASmcP6KBAXn9vT4MZF9pBPMI9icocvugk2A7E9b8zr_INUK-ZOyZYGxaK3AFs0sEqv6siqpedORTPM1cQONGZCAhTmfZtJBESEACgjZ4V0C5Vdwzde3ZRg-AR-_E4uzalM3-EFDe7r/s1600/Screen+Shot+2016-12-05+at+8.55.42+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgS6EASmcP6KBAXn9vT4MZF9pBPMI9icocvugk2A7E9b8zr_INUK-ZOyZYGxaK3AFs0sEqv6siqpedORTPM1cQONGZCAhTmfZtJBESEACgjZ4V0C5Vdwzde3ZRg-AR-_E4uzalM3-EFDe7r/s400/Screen+Shot+2016-12-05+at+8.55.42+AM.png" width="270" /></a></div>
<div>
Gartner provides customer reviews of many of these vendors <a href="https://www.gartner.com/reviews/market/business-intelligence-analytics-platforms?refval=mq_reprint&campaign=mq_reprint&content=BI_Analytics">here</a>.<br />
<br /></div>
<h4>
Gartner Magic Quadrant</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRB7ad0Gjt6ktvkEFpZbRkcunoKRDMuW_oS2V1SzGgoHLnGOb8jCJzdr9tjxobbrNEEzFTnZ6-tMxM5m5JxehSMxi798NqwKMeiRRbp2lQx4xJFtkABF0HMhvQT1x3dxzb0gvu4PqyOplZ/s1600/Screen+Shot+2016-12-05+at+9.16.04+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRB7ad0Gjt6ktvkEFpZbRkcunoKRDMuW_oS2V1SzGgoHLnGOb8jCJzdr9tjxobbrNEEzFTnZ6-tMxM5m5JxehSMxi798NqwKMeiRRbp2lQx4xJFtkABF0HMhvQT1x3dxzb0gvu4PqyOplZ/s400/Screen+Shot+2016-12-05+at+9.16.04+AM.png" width="397" /></a></div>
<div>
<br />
Note that the Leaders rank only "fair" on average across Gartner's 14 critical capabilities. Of those with an average rank of "good" most are in the Visionary quadrant and 45% in Niche. Our deep pocket product vendors are mostly average "fair" as well. Interesting market right now.<br />
<br /></div>
<h4>
Something to consider: What do you really want? </h4>
<div>
Gartner assessment criteria are somewhat conservative focusing as much on vendor stability, size etc. as on the quality of the underlying product implementation. To some extent that unmeasured quality characteristic dictates the vendor's ability to innovate and deliver capability differentiation. This suggests Gartner may be a poor guide if you are after differentiation. For example, graph processing and databases as well as graph based analytics delivered via Cloud shows promise. Look at what <a href="https://www.facebook.com/notes/facebook-engineering/scaling-apache-giraph-to-a-trillion-edges/10151617006153920/">Facebook</a>, <a href="https://neo4j.com/blog/ebay-competitive-advantage-neo4j/">Ebay</a>, AWS and Google have been up to. More on this in a future post. Another example is in the Data Visualization space. Let's take a quick look there now.</div>
<div>
<br /></div>
<h4>
Data Visualization Product Landscape</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpNkaO5TFJkc4hN8dSF_9NTe3ZMzdPqKQ82no4AwcOWqjMDiIlTAPFKoI9GLI3hxmoX_4RmUH1VahcjDPixgDV7nkHJK0HNqGeGgUgu_ZClDfmNC9Jjkea0UN95F9BMFaO_eMpffQbdV3f/s1600/Screen+Shot+2016-12-05+at+9.19.29+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="252" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpNkaO5TFJkc4hN8dSF_9NTe3ZMzdPqKQ82no4AwcOWqjMDiIlTAPFKoI9GLI3hxmoX_4RmUH1VahcjDPixgDV7nkHJK0HNqGeGgUgu_ZClDfmNC9Jjkea0UN95F9BMFaO_eMpffQbdV3f/s400/Screen+Shot+2016-12-05+at+9.19.29+AM.png" width="400" /></a></div>
<div>
<br /></div>
<div>
<a href="https://en.wikipedia.org/wiki/Data_visualization#References">Data Visualization</a> is key to modern Analytics. Roughly half of the top products made the Gartner list. Again - makes sense given Gartner's assessment criteria - but it sure seems there is opportunity out there to leverage!</div>
<div>
<br /></div>
<div>
<br /></div>
<h4>
Additional References</h4>
<div>
Here is link to the <a href="https://docs.google.com/spreadsheets/d/112YrhQXZWa9cY-eB1gcxejRylMFQvkBbUcy3-GXqxYQ/edit?usp=sharing">google sheet</a> used to do the analysis above. It has a bit more data and links to more as well.</div>
<div>
<br /></div>
<div>
<br /></div>
AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com16tag:blogger.com,1999:blog-5654895526745383189.post-65121732844679436052016-11-25T09:30:00.000-05:002016-12-23T10:00:34.173-05:00Kanban: Successful Evolutionary Change for Your Technology BusinessKanban: <a href="https://www.amazon.com/Kanban-Successful-Evolutionary-Technology-Business/dp/0984521402">Successful Evolutionary Change for Your Technology Business</a> by David Andersen.<br />
<br />
If you are an Agilist then this is a must read book. This is another case of my experiencing key concepts from a book prior to reading it. The importance of slack, focussing on cycle time, empowering teams...<br />
<br />
I wish I would have read this the day it came out. I'm working on a summary now. It starts <a href="http://agilejazz.blogspot.com/p/kanban-book-summary.html">here</a>.AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com2tag:blogger.com,1999:blog-5654895526745383189.post-70868377374510558682016-11-18T15:10:00.001-05:002016-11-18T16:40:47.821-05:00DevOps Business Cases - Chatting with Robert...I had a chance to catch up with my friend Robert Boyd. We had a nice chat about DevOps business cases. Prior to talking I threw together a Google Slides deck to have something for us to react to. It was a very nice chat and it turns out we were seeing things very much the same way. We made a couple of tweaks to the ideas as we talked, namely we added Robert's favorite three top down metrics at the end.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEha6DrNIiPcxr74rUCoPh66q488OdhdsgLNDHC6XrMlHRYl0wMmwEMpOtbD4apzMjxW3G01LYKrHKv2RTacwgQlkr7JynFV1vRl53W9AACSMvWOQiKqHY4FF-stXlgp2Je-0zoBiSDWArrh/s1600/Screen+Shot+2016-11-18+at+2.45.48+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="180" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEha6DrNIiPcxr74rUCoPh66q488OdhdsgLNDHC6XrMlHRYl0wMmwEMpOtbD4apzMjxW3G01LYKrHKv2RTacwgQlkr7JynFV1vRl53W9AACSMvWOQiKqHY4FF-stXlgp2Je-0zoBiSDWArrh/s320/Screen+Shot+2016-11-18+at+2.45.48+PM.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The deck would not imbed in a way that looked nice so this post has the content from the deck in simple outline form:</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<h3 style="clear: both; text-align: left;">
Top Down DevOps Benefits</h3>
<div>
<ol>
<li>Reduce time to market - realize value faster, less coordination costs</li>
<li>Hypothesis test value and growth assumptions using actual customer behavior, improve feature ROI</li>
<li>Reduce/eliminate production incidents from changes, faster mean time to recover (MTTR): avoid costs, build customer goodwill</li>
<li>Reduce/repurpose operating costs- automate tasks across development, infrastructure and operations</li>
</ol>
</div>
<h3>
DevOps Levers</h3>
<div>
<ol>
<li>Establish a Lean, self-improving culture</li>
<li>Loosely coupled, intrinsically testable architectures, applications and services that are easy to change and scale</li>
<li>Automated, self-service build pipelines of production like environments reduce environmental errors/defects</li>
<li>Automated test suites with high test coverage to reduce defects</li>
<li>Automated deployments to production reducing opportunities for error</li>
<li>Enhanced telemetry tied to change events and self-service reporting reveal cause and effect thus improve MTTR</li>
<li>Release automation and controls e.g. feature toggles, canary, blue green enabling flexible releases</li>
<li>Split testing capabilities enabling customer hypothesis testing</li>
</ol>
</div>
<h3>
DevOps Enablers </h3>
<h4>
Organizational models</h4>
<div>
<ul>
<li>*Market based (value stream aligned) integrated teams (infra, ops, dev, test, product)</li>
<li>Functional (capability aligned, e.g. dev, ops, test, platform) deeply integrated lean process e.g. test first, ops requirements defined up front, phased production transition to run, platform team owns shared self-service capabilities</li>
<li>Matrix - mix of above two</li>
</ul>
</div>
<h4>
Practices</h4>
<div>
<ul>
<li>TDD/ATDD (XP)</li>
<li>Scrum/Kanban</li>
<li>Lean analysis e.g. A3, Value Stream Analysis Five Why’s</li>
<li>Statistical analysis of operations</li>
<li>...</li>
</ul>
</div>
<h4>
Technologies</h4>
<div>
<ul>
<li>Cloud</li>
<li>Infrastructure as code</li>
<li>Continuous Integration</li>
<li>Deployment and release automation</li>
<li>Cluster management (e.g. improve utilization via oversubscription) </li>
<li>...</li>
</ul>
</div>
<h4>
Three Top Down Metrics</h4>
<div>
Focus whole organization on a shared goal, improve:</div>
<div>
<ul>
<li>Deploys/day</li>
<li>Prod issues/month</li>
<li>MTTR</li>
</ul>
</div>
<div>
These metrics create a punch list of inefficiencies in the system to attack via solid monitoring, test coverage, sustainable architectures etc.</div>
<div>
<br /></div>
<div>
Map the value stream, identify and subordinate constraints...</div>
<div>
<br /></div>
<div>
--------------------------------------------------------------------------------------------------------------------------</div>
<div>
If you'd like to see the deck it's here: <a href="https://docs.google.com/presentation/d/1oQmRTpcDH3jcBi9lqWzrBc5DA6EjbIXiLqGOKTFQ1O0/edit?usp=sharing">DevOps Business Case Chat</a> (looks nicer - I like Google Slides new "explore" formatting feature)...</div>
AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com28tag:blogger.com,1999:blog-5654895526745383189.post-59849377974629100462016-11-18T09:13:00.000-05:002016-12-23T09:32:42.678-05:00The DevOps HandbookWhile we didn't call it DevOps back in 2012, when I joined Gap Inc.,we working towards achieving some of its goals then. Over the ensuing years we got reasonably good at it. I read Gene Kim's book, The Phoenix Project, a couple years back and so was excited to learn he was working on a new book entitled The DevOps Handbook.<br />
<br />
I pre-ordered the book on Amazon and wrote a <a href="http://agilejazz.blogspot.com/p/the-devops-handbook-summary-1-of-4.html">four part summary</a> as I read it. This is as must read book if your in IT today. The book is long so if you are in a hurry and want to read my take on the most important concepts the summary is for you. It also serves as a good review if you've read it and just want to keep it in your head.AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com6tag:blogger.com,1999:blog-5654895526745383189.post-51703376194252523592016-11-15T11:37:00.001-05:002016-12-05T09:59:02.868-05:002016 DevOps Vendor LandscapeI've been curious about the various DevOps tool vendors, beyond those my team's have used. So here is some research that I'd like to share. We'll start with the 2016 Gartner Magic Quadrant report and then add some to it. There's a link to my google sheet with <i>quick</i> vendor comparisons and a link to the Gartner report too.<br />
<br />
<h4>
Gartner 2016 Magic Quadrant for Application Release Automation:</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPUIvwXlod1FPQScYx01gzWZ7KRhLxVLcYYY3PEU6za0NOjzKSTJL1VJDRIlCRCkVVS4ZFgVGIN3n_3KXVuZjTZoXlqN2kkN-liDl6rFuRgUcVJ2lR3BNQA_jzxsm2HdV7WAfijGDz1pnB/s1600/Screen+Shot+2016-11-15+at+6.49.06+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="393" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPUIvwXlod1FPQScYx01gzWZ7KRhLxVLcYYY3PEU6za0NOjzKSTJL1VJDRIlCRCkVVS4ZFgVGIN3n_3KXVuZjTZoXlqN2kkN-liDl6rFuRgUcVJ2lR3BNQA_jzxsm2HdV7WAfijGDz1pnB/s400/Screen+Shot+2016-11-15+at+6.49.06+AM.png" width="400" /></a></div>
<br />
For the most part the leaders quadrant is dominated by the traditional players - IBM, Automic, CA Technologies each with >30 years experience. They've done a mix of building and acquiring capabilities. Two younger players Electric Cloud and Xebia Labs are both venture backed, private firms.<br />
<br />
My initial preferences in the leaders quadrant are IBM and Xebia Labs. They are both pretty end to end. IBM's stateless architectural approach and Xebia labs somewhat unique agentless architecture make them jump out for me.<br />
<br />
The visionary quadrant's sole member is Clarive. However, their implementation seemed convoluted. Hashicorp's potentially a visionary though. While they are open source and thus not for everyone their architectural vision resonates with me. Their immutable infrastructure philosophy and agent and agentless capabilities stand out. I couldn't figure out their workflow paradigm in the time I spent looking. They are too small to make the list at this point - but are worth looking at, especially if your comfortable in the open source world. Studying them may highlight relevant problems everyone's solving which are not often discussed in the big players marketing collateral.<br />
<br />
Serena Software is the sole player in the Challengers quadrant. While their offering is broad its also really heavy. They focus on highly regulated industries.<br />
<br />
The stand out in the Niche quadrant for me is Puppet. Their test centric philosophy bodes well for them as they build out their offerings.<br />
<br />
Atlassian - also not on the list - deserves special mention. If your interested in piecing together your own solution and your a development centric organization they may be a good fit. They also provide and opportunity to call out the smaller specialty players like Ansible, RunDeck, Chef etc. There's a nice <a href="http://blogs.atlassian.com/2014/04/practical-continuous-deployment/">blog entry</a> written by one of their Directors that is a good DevOps read.<br />
<br />
Finally, a word about workflow. My experience suggests we be wary of over subscribing to the "drag and drop" philosophy. Xebia labs sums it up well in their <a href="https://xebialabs.com/assets/files/whitepapers/5-reasons-stay-away-workflows.pdf">5 Reasons to Stay Away from Workflows</a> technical note. Bottom line, DevOps at scale is complex. Dealing effectively with that complexity determines your level of success. Workflow can be leveraged as a tool to help visualize the broad strokes but becomes a spaghetti mess if you target "reuse" and assume "making it visible" allows people who don't understand software development to manage the complexity of DevOps implementations at scale.<br />
<br />
Here's the google sheet with the high level comparisons and some comments on all of these and a few more players: <a href="https://drive.google.com/open?id=1l4h2cbWAoaLzRtbVJtwcgUJGlkd6BoNGPvbrlUL8s_w">DevOps Players Nov 2016</a>. <br />
<br />
Here's the <a href="https://drive.google.com/open?id=0B6G9zKtMuGNMUUFsbjVIdnV3enc">Gartner report</a> too.<br />
<br />
Enjoy!AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com154tag:blogger.com,1999:blog-5654895526745383189.post-21374162276146686902016-10-14T08:54:00.001-04:002022-03-29T11:13:13.192-04:00Badass: Making Users AwesomeI learned of Badass: Making Users Awesome by Kathy Sierra after Bob Payne referred me to the Modern Agile movement. Kathy's book seeks to answer one question: Given competing equally priced, equally promoted products or services, why are some far more successful than others?<br />
<br />
This <a href="http://agilejazz.blogspot.com/p/badass-making-users-awesome.html">400 word summary</a> will give you a sense of the book. If you create products for a living this then I highly recommend this book!AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com1tag:blogger.com,1999:blog-5654895526745383189.post-42483310145370673252016-09-16T08:45:00.000-04:002016-12-23T09:32:42.674-05:00The Lean Start UpThe Lean Start Up: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses by Eric Ries.<br />
<br />
While working at Gap Inc. concepts from this book were frequently leveraged. I learned by doing. After leaving Gap I read the book and see we implemented some of the concepts well and others, less so.<br />
<br />
This is a great book worthy of study. I created a <a href="http://agilejazz.blogspot.com/p/lean-start-up-excerpt-based-summary.html">book summary</a> using my kindle highlights. The summary is almost entirely excerpts from the book. My goal was to create a reference or study guide for myself and to share with others. If you haven't read the book and don't have the time to read the whole book (takes maybe a few days) then this summary is worth your time to read. If you only want to review the key concepts - it is probably not for you.<br />
<br />
If I had it to do over again I would have synthesized more and excerpted less. Live and learn.<br />
<br />
<br />
<br />AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com1tag:blogger.com,1999:blog-5654895526745383189.post-72654783081246747382016-09-09T10:20:00.000-04:002016-12-23T11:57:26.512-05:00Agile Leadership Video InterviewMy friend <a href="http://lithespeed.com/speaker/bob-payne/">Bob Payne</a> from <a href="http://lithespeed.com/">LitheSpeed</a> who does the <a href="http://agiletoolkit.libsyn.com/">Agile Toolkit Podcast</a> series interviewed me yesterday. The topic was Agile Leadership and he published the video interview on LitheSpeed's youtube channel so I thought I'd share. It was an engaging talk with Bob about agile leadership.<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/spbK2jbpxCc" width="560"></iframe>AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com1tag:blogger.com,1999:blog-5654895526745383189.post-35342890548908388132016-08-19T08:13:00.000-04:002016-12-23T08:14:13.044-05:00Lean IT Field GuideA colleague from my Nationwide days, Tom Paider, co-authored a book with Michael Orzen called The Lean IT Field Guide: A Roadmap for Your Transformation. The book based in part on our shared experience of establishing Nationwide's Application Development Center. More on that <a href="http://www.bizjournals.com/columbus/print-edition/2010/10/29/nationwide-turns-in-house-as-part-of-a.html">here</a>.<br />
<br />
His book is a how to guide for applying certain aspects of Lean to corporate IT. The focus of the book is summarized below:<br />
<br />
<span style="font-family: inherit;">The modern enterprise has a simple goal: create sustainable value i.e. make money now and in the long run. A principle challenge is leading and managing a very large number of people to achieve the goal. To achieve the goal most would agree we need:</span><br />
<br />
<ol>
<li><span style="font-family: inherit;">Alignment from the top down on purpose and strategy</span></li>
<li><span style="font-family: inherit;">Effective bottom up execution matched to the strategy</span></li>
<li><span style="font-family: inherit;">Clear feedback enabling the adjustments needed to achieve the purpose</span></li>
</ol>
<br />
<span style="font-family: inherit;">In brief Lean enables people at every level to understand and improve the part they play in achieving the goal. Employees define baseline work processes called "standard work," along with their performance metrics, problems and potential solutions called "counter-measures." Collaborative cycles of Plan, Do, Check, Adjust apply the scientific method to solve these problems using a simple "A3" document. Leaders "Go See" on "Gemba Walks" to understand what's happening and to mentor and learn how to support their people. All of this is big and visible. All of this happens at every level - not just the front line.</span><br />
<div>
<br /></div>
<div>
I created an excerpt based summary of the book in two parts. Part one starts <a href="http://agilejazz.blogspot.com/p/lean-it-field-guide-1-of-2.html">here</a>.</div>
AgileJazzhttp://www.blogger.com/profile/10875588744299916024noreply@blogger.com1