FEED Validator

for Atom and RSS and KML

Congratulations!

This is a valid Atom 1.0 feed.

Recommendations

This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.

line 5, column 88: Self reference doesn't match document location [help]

... el="self" type="application/atom+xml" />
                                             ^

line 454, column 0: Non-html tag: t (3 occurrences) [help]

&lt;p&gt;I had long since adopted a practice of creating discrete configurat ...

Source: https://feeds.feedburner.com/LosTechies

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" >
<generator uri="https://jekyllrb.com/" version="3.9.2">Jekyll</generator>
<link href="https://lostechies.com/feed.xml" rel="self" type="application/atom+xml" />
<link href="https://lostechies.com/" rel="alternate" type="text/html" />
<updated>2022-11-29T03:33:34+00:00</updated>
<id>https://lostechies.com/</id>
<title type="html">Los Techies</title>
<subtitle> </subtitle>
<entry>
<title type="html">Pragmatic Deferral</title>
<link href="https://lostechies.com/derekgreer/2022/05/31/pragmatic-deferral/" rel="alternate" type="text/html" title="Pragmatic Deferral" />
<published>2022-05-31T13:00:00+00:00</published>
<updated>2022-05-31T13:00:00+00:00</updated>
<id>https://lostechies.com/derekgreer/2022/05/31/pragmatic-deferral</id>
<content type="html" xml:base="https://lostechies.com/derekgreer/2022/05/31/pragmatic-deferral/">Software engineering is often about selecting the right trade offs. While deferring feature development is often somewhat straight-forward, based upon a speculation about the return on investment, and generally decided by the customer; marketing; sales; or product people; low-level implementation decisions are typically made by the development team or individual developers and can often prove to be a bit more contentious among teams with a plurality of strong opinions. This is where principles like <a href="https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it">YAGNI</a> (You’re Aren’t Going to Need It), or the <a href="https://en.wikipedia.org/wiki/Rule_of_three_(computer_programming)">Rule of Three</a> have often been set forth as a guiding heuristic.
While I generally advise the teams I coach to allow the executable specifications (i.e. the tests) to drive emergent design and to defer the introduction of ancillary libraries, frameworks, patterns, and custom infrastructure, until you need it, there is a level of pragmatism that I employee when determining when to introduce such things.
I’ve been a fan of Test-Driven Development for some time now and have practiced it for over a decade. One of the primary benefits of Test-Driven Development is having an objective measure guiding what needs to get built. For example, if the acceptance criteria for a User Story concerns building a new Web API for a company’s custom B2B solution, your specs are going to drive out some sort of HTTP-based API. What the specs won’t dictate, however, are decisions such as whether to use an MVC framework, an IOC container, whether to introduce a fluent validation library or an object mapping library. Should we adhere strictly to principles like <a href="https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it">YAGNI</a> or the <a href="https://en.wikipedia.org/wiki/Rule_of_three_(computer_programming)">Rule of Three</a> for guidance here? My answer is: it depends.
Deferring software decisions comes with quite a range of consequences. Some decisions, such as whether to select ASP.NET MVC at the outset of a .Net-based Web application, could cause quite a bit of rework if you were to defer such a decision until working with lower-level components started to reveal friction or duplication. Other decisions, such as deferring the introduction of an object mapping library (e.g. Automapper) until the shape of the objects you’re returning actually differ from your entities essentially have only positive consequences. But how do we know?
The YAGNI principle is very similar to the firearm safety rule “The Gun is Always Loaded”. No, the gun isn’t always loaded … but it’s best to treat it like it is. Similarly, “You aren’t going to need it” doesn’t really mean you may not need it, but it’s intended to help you avoid unnecessary work. That is, until it causes more work.
In software engineering, the more you code, the more you’ll have to maintain. The Art of Not Doing Stuff, when correctly applied, can save companies as much or more money than building the right things. While I’m not religious these days, there’s a definition of the term “Hermeneutics” that I heard years ago from a Christian radio personality, Hank Hanegraaff. He would say: “Hermeneutics is the art and science of biblical interpretation”. He would go on to explain, it’s a science because it’s guided by a system of rules, but it’s an art in that you get better at it the more you do it. Having heard that explanation years ago, I have long felt these properties are equally descriptive of software development.
For myself, I take a pragmatic approach to YAGNI in that I make selections for a number of things at the outset of a new project which I’ve recognized, through experience, have resulted in less friction down the road; and I defer choices which I reason to have little to no cost by implementing at the point a given User Story’s acceptance criteria drives the need. For example, I do start off setting up a Web project using ASP.NET MVC. I do set up end-to-end testing infrastructure. I do add an open source DI container and set up convention-based registration. These are things which I’ve found actually cause me more friction if I pretend I’m not going to need them. I don’t want to implement my own IHttpHandler and wait until I see the need for a robust routing and pipeline framework and have to go back and reimplement everything. I don’t want to be hand-rolling factories over and over and have to go back and modify code at the point enough duplication reveals the need for dependency injection, and I don’t want to edit a Startup.cs or other bootstrapper component each time a component has a new dependency. Outside of these few concerns, however, I do typically defer things until needed.
Ultimately, this pragmatism isn’t an exception to the YAGNI rule so much as it is a judicial application of YAGNI within a larger strategy of practicing <a href="https://agilemanifesto.org/principles.html">the art of maximizing the amount of work not done</a>. In short, apply YAGNI when it makes you more agile, not less.</content>
<author>
<name>Derek Greer</name>
</author>
<summary type="html">Software engineering is often about selecting the right trade offs. While deferring feature development is often somewhat straight-forward, based upon a speculation about the return on investment, and generally decided by the customer; marketing; sales; or product people; low-level implementation decisions are typically made by the development team or individual developers and can often prove to be a bit more contentious among teams with a plurality of strong opinions. This is where principles like YAGNI (You’re Aren’t Going to Need It), or the Rule of Three have often been set forth as a guiding heuristic.</summary>
</entry>
<entry>
<title type="html">Magical Joy</title>
<link href="https://lostechies.com/derekgreer/2022/05/27/magical-joy/" rel="alternate" type="text/html" title="Magical Joy" />
<published>2022-05-27T13:00:00+00:00</published>
<updated>2022-05-27T13:00:00+00:00</updated>
<id>https://lostechies.com/derekgreer/2022/05/27/magical-joy</id>
<content type="html" xml:base="https://lostechies.com/derekgreer/2022/05/27/magical-joy/">In a segment of an interview with host Byron Sommardahl on <a href="https://podcasts.apple.com/us/podcast/all-things-senior-derek-greer/id1584867029?i=1000541910261">The Driven Developer Podcast</a>, recorded in the summer of 2021, Byron and I discussed a bit about a pattern I introduced to our project when we worked together in 2010 which Byron later dubbed “The Magical Joy Bus” 😂. That pattern was the <a href="https://stackoverflow.com/a/65295855/1219618">Command Dispatcher</a> pattern. We unfortunately didn’t have the time I would have liked to fully unpack my thoughts and experiences with using this pattern over the years, so I thought I’d share that here.
In brief, the Command Dispatcher pattern is one where a central component is used to decouple a message issuer from a message handler. Many .Net developers have become familiar with this pattern through Jimmy Bogard’s open source library: <a href="https://www.nuget.org/packages/MediatR/">MediatR</a>. While I’ve never personally used the MediatR library, I have used a far more simplistic implementation throughout the years. Since my implementation was only ever primarily a single class, I never felt particularly motivated to release it as an open source library. I did, however, share my code with a former colleague a few years ago who has since packaged up a slightly modified version of my original <a href="https://github.com/joelbrinkley/dispatch">here</a>.
Back in 2010 and the following years, my motivation for using the pattern within the context of .Net Web applications was primarily to write clean controller actions, facilitate adherence to the Single Responsibility Principle within the Application Layer, and to eliminate the need for injecting extraneous controller or Application Service dependencies. For earlier versions of ASP.Net MVC, I still see it as a worthwhile pattern to implement. It certainly, however, has its drawbacks.
As alluded to by Byron in my interview, the team I was working with back then didn’t quite like the “magic” involved with the design. The primary issue for my teammates was that you couldn’t easily navigate from a controller action to the message handler directly via Visual Studio’s “Edit.GoToDefinition” (i.e. F12) shortcut. This was an unfortunate shortcoming of this approach, but one over which I never experienced a large degree of angst as it was in essence no different than the process one must go to in locating a controller action being invoked as the result of a given Web request. All convention-over-configuration approaches suffer from some degree of degradation in discoverability and navigation. Of course, the frequency in which developers find themselves needing to navigate from controllers to components within an Application layer is really where the issue lies.
We didn’t get around to discussing Byron’s intuition about the design all those years ago in the podcast, but Byron and my former colleagues weren’t alone in how they felt about the pattern. Over the years, I introduced the pattern to two other teams, both of which expressed some of the same feelings of disdain over its impact on stepping through the code. Eventually, I came to the conclusion that, while I still saw the same benefits in the pattern’s implementation, there really was just too much friction in getting teams on board with its adoption.
Fortunately with the advent of .Net Core which introduced the [FromServices] attribute, we can achieve the same benefits mentioned earlier by injecting handlers directly into Controller Actions:
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> [HttpGet]
public async Task&lt;IActionResult&gt; GetWidgets([FromQuery] GetPaginatedWidgetsRequest request, [FromServices] GetWidgetsRequestHandler handler)
{
return await handler.Handle(request).ToResult(r =&gt; new OkObjectResult(r), r =&gt; BadRequest());
}
</code></pre></div></div>
This is my preferred approach today. While it allows us to keep our controllers clean; to write small, focused Application Layer handler classes; and to avoid injection of unused dependencies; it’s also easy for developers at any level to work with and maintains the standard navigation and debugging experience. Win-win!</content>
<author>
<name>Derek Greer</name>
</author>
<summary type="html">In a segment of an interview with host Byron Sommardahl on The Driven Developer Podcast, recorded in the summer of 2021, Byron and I discussed a bit about a pattern I introduced to our project when we worked together in 2010 which Byron later dubbed “The Magical Joy Bus” 😂. That pattern was the Command Dispatcher pattern. We unfortunately didn’t have the time I would have liked to fully unpack my thoughts and experiences with using this pattern over the years, so I thought I’d share that here.</summary>
</entry>
<entry>
<title type="html">User Stories</title>
<link href="https://lostechies.com/derekgreer/2022/05/25/user-stories/" rel="alternate" type="text/html" title="User Stories" />
<published>2022-05-25T13:00:00+00:00</published>
<updated>2022-05-25T13:00:00+00:00</updated>
<id>https://lostechies.com/derekgreer/2022/05/25/user-stories</id>
<content type="html" xml:base="https://lostechies.com/derekgreer/2022/05/25/user-stories/">The use of User Stories has become fairly commonplace in the software industry. First introduced as an agile requirements-gathering process by Extreme Programming, User Stories arguably owe their popularity most to the adoption of the Scrum framework for which User Stories have become the de facto expression of its prescribed backlog.
So what exactly is a User Story? Put simply, they are a light-weight approach to expressing the desired needs of a software system. The idea behind User Stories, which was introduced as simply “Stories” in the book Extreme Programming Explained - Embrace Change by Kent Beck, was to move away from rigid requirements gathering processes in process, form, and nomenclature. Beck explained that the very word “requirement” was an inhibitor to embracing change because of its connotations of absolutism and permanence. At their inception, the intended form of stories was to create an index card containing a short title, simple description written in prose, and an estimation.
<h2 id="the-three-part-template">The Three-Part Template</h2>
In the late 1990’s, a software company named Connextra was an early adopter of Extreme Programming. In contrast to the distinct roles defined by the Scrum framework, XP doesn’t prescribe any specific roles, but is intended to adapt to existing roles within an organization (e.g. project managers, product managers, executives, technical writers, developers, testers, designers, architects, etc.).
The origin of most of Connextra’s stories were from members of their Marketing and Sales departments which wrote down a simple description of features they desired. This posed a problem for the development team, however, for when the time came to have a conversation about the feature, the development team often had difficulty locating the original stakeholder to begin the conversation. This led the team to formulate a 3-part template to help address friction resulting from ambiguous requirement sources. Their 3-part template is as follows:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> As a [type of user]
I want to [do something]
So that I can [get some benefit]
</code></pre></div></div>
Ironically, while the 3-part template has become the defacto standard for authoring User Story descriptions, Scrum’s “Product Owner” role, most often filled by product development specialists acting as customer proxies, along with the use of software agile-planning tools such as Confluence, Planview, Azure DevOps Boards, etc., which captures who created a given story, tends to greatly diminish the need from which the template originated. This template has since become quite the de facto standard in expressing User Story Descriptions. The irony is that many teams, in caro-cult fashion, often utilize the 3-part template where the original need to identify the author of the story to start the conversation no longer exists. Change has occurred, but because many didn’t understand the underlying impetus for the 3-part template, they were incapable of adapting to that change.
Jeff Patton writes the following concerning the prevalent use of the 3-part story template in his book “User Story Mapping”:
<blockquote>
“… the template has become so ubiquitous, and so commonly taught, that there are those who believe that it’s not a story if it’s not written in that form. … All of this makes me sad. Because the real value of stories isn’t what’s written down on the card. It comes from what we learn when we tell the story.”
</blockquote>
Mike Cohn, author of many books on agile processes including “User Stories Applied” and “Agile Estimating and Planning” writes similarly:
<blockquote>
“Too often team members fall into a habit of beginning each user story with “As a user…” Sometimes this is the result of lazy thinking and the story writers need to better understand the product’s users before writing so many “as a user…” stories.”
</blockquote>
Cohn’s observations are spot on. In my experience, not only does this happen “too often”, it’s the rule, not the exception. It’s really just human nature. The moment a process becomes formulaic, teams will begin to just go through the motions without engaging their minds. This can be good for manual tasks like brick-laying, or cleaning a house, but it is detrimental to processes intended to promote communication. Sadly, many teams spend an inordinate amount of time on the trappings of things like ensuring their requirements follow the 3-part story template rather than using the story as a tool for its original intent: A placeholder for a conversation.
<h2 id="there-and-back-again">There and Back Again</h2>
While not explicitly stated, the original idea behind Stories in Extreme Programming was to facilitate a conversation, not to define an objective goal. The agile movement started as a way to address issues in the industry’s largely failing attempts to apply manufacturing processes to software development. In particular, Stories were intended to address the underlying motivation for requirements (i.e. how teams determine what to build), not to themselves be requirements.
In many ways, today’s User Stories have become the antithesis of what Kent Beck originally intended. Sadly, much of what is marketed as “agile” today has been corrupted by traditional-minded business analysts, product managers, and marketing agencies who never really understood the agile movement fully. User Stories have, to a large extent, become a casualty of these groups. We’ve gone from requirements to stories and back again. As described by Jeff Patton, “Stories aren’t a way to write better requirements, but a way to organize and have better conversations.”
<h2 id="the-better-way">The Better Way</h2>
Ultimately, the question companies seek to answer is: How do we determine the features which provide the best ROI for the business? While it may seem counterintuitive to some, customers aren’t generally the best source for determining what features to build. They can be a source, but they aren’t generally a team’s best source. Customers are, however, the best source for determining how customers currently work, what problems they face, and what friction is involved in any current processes. Various analysis techniques can be used to solicit customer opinions on desired features, but it’s best to rely upon such techniques merely as means to distill the problems currently faced by customers. From there, stories are best created with a simple title and a description of the customer’s problem written in prose with the intent for the description to serve as a starting point for a conversation with the team.
The best way to determine what to build is as a member of a mature agile team. The operative word here is mature. What makes for a mature team is a Product Owner with a background in the problem domain space, a Team Coach with deep knowledge of agile and lean processes, and 3-5 cross-functional developers weighted toward senior experience who have gone through a forming, storming, norming, and performing phase.
User Stories shouldn’t be feature requests, but rather a placeholder for a conversation. A conversation with whom? With your team. About what? About how to iteratively solve the problems you learned from customers in small steps with frequent feedback. Product Owners should not bring requirements to a development team. There’s great power in collaboration. A smart team of 5 to 7 individuals including a subject matter expert (what the Product Owner should bring to the table) and a coach are a far better source for what features to build than just the customer or the Product Owner.
<h2 id="an-example">An Example</h2>
The following is an example story which more closely follows the original intent of Stories.
Our scenario involves a company which provides a website allowing customers to create wedding and gift registries to send to others. In its current form, the site allows customers to pick from among existing vendors, but the company frequently receives requests from customers about specific products they’d like to see included. The current process involves the Sales team creating tickets for their Operations team to add new vendors to the site which involves updating the production database directly. Additionally, the work currently falls to one person whose job entails other operation tasks which often results in a delay to the timely fulfillment of customer requests.
The following represents the story:
<table style="border: 1px solid black; background-color: white; color: black">
<tr>
<td>
<h2>Easily Manage Registry Products</h2>
<hr style="border-top: 1px solid black" />
<h3>Description</h3>
Our customers often want to add products that aren't part of our current vendor product list. This causes the sales team to constantly have to put in tickets and currently Margret is the only one that is working the tickets. We need a better solution!
</td>
</tr>
</table>
Note how the description is written in prose (i.e. in normal conversational language), and doesn’t follow the wooden 3-part template. Note also, the story doesn’t prescribe how to solve the problem. It just provides background on what the problem is and who it affects. It isn’t just that the story doesn’t dictate implementation details, but that it doesn’t dictate the solution at all. This is the ideal starting point for most stories. It’s a placeholder for a conversation about how to solve the problem.
From here, the team would collaborate on the story to determine the best solution that results in the smallest feature increment which adds value to the end user. Several ideas may be discussed. The system could integrate with a 3rd-party content management system, allowing people within the company without SQL experience to update content. Alternately, the team may decide that adding a feature to allow customers to add custom products directly to their personal event registry is both easier, and scales far better than solutions requiring company employees to work tickets.
As part of a story refinement session, the team may update the story with acceptance criteria to guide the implementation:
<table style="border: 1px solid black; background-color: white; color: black">
<tr>
<td>
<h2>Easily Manage Registry Products</h2>
<hr style="border-top: 1px solid black" />
<h3>Description</h3>
Our customers often want to add products that aren't part of our current vendor product list. This causes the sales team to constantly have to put in tickets and currently Margret is the only one that is working the tickets. We need a better solution!
 
<h3>Acceptance Criteria</h3>
When the customer navigates to the edit registry view 
&nbsp;&nbsp;it should contain a link for adding custom products
 
When the customer clicks the add custom product link 
&nbsp;&nbsp;it should navigate to the add custom product view (note: see balsamiq wireframe attached)
 
When the customer adds a new custom product with valid inputs 
&nbsp;&nbsp;it should add the custom product to the customers registry 
&nbsp;&nbsp;it should display a success message in the application banner 
&nbsp;&nbsp;it should navigate back to the edit registry page
 
When the customer enters invalid custom product parameters 
&nbsp;&nbsp;it should show standard field level error messages 
&nbsp;&nbsp;it should not enable the save button 
</td>
</tr>
</table>
While an Acceptance Criteria section isn’t mandatory, it can often be valuable for helping to frame the scope of the story, a reminder to the team of the high-level plans discussed for deferred work, and/or may serve as the team’s Definition of Done. For small teams involving just a few members, or for highly adaptive and collaborative teams, it may be enough to just just write “We decided to add a feature to allow the customer to add their own products!”. The team may very well take the initial story description and rapidly iterate on a solution, deciding together when they think it’s done! (Gasp!) Of course, this level of informality probably is only best suited to highly cohesive, highly functioning teams. For inexperienced to moderately experienced teams, some denotation of Acceptance Criteria would be advisable. The key point is, the story didn’t arrive to the team in the form of requirements, but as a placeholder for a conversation.
<h2 id="conclusion">Conclusion</h2>
As the adoption of agile frameworks such as Scrum have become more mainstream, a number of practices have become formulaic and adopted by teams via a cargo-cult onboarding to agile practices without truly grasping what it means to be agile. The User Story has all but lost it original intent by many teams who have done little more than slap agile labels onto Waterfall manufacturing processes. User Stories were never intended to be requirements, but rather a placeholder for a conversation with the development team. Let’s do better.</content>
<author>
<name>Derek Greer</name>
</author>
<summary type="html">The use of User Stories has become fairly commonplace in the software industry. First introduced as an agile requirements-gathering process by Extreme Programming, User Stories arguably owe their popularity most to the adoption of the Scrum framework for which User Stories have become the de facto expression of its prescribed backlog.</summary>
</entry>
<entry>
<title type="html">.Net Project Builds with Node Package Manager</title>
<link href="https://lostechies.com/derekgreer/2020/12/10/dotnet-project-builds-with-npm/" rel="alternate" type="text/html" title=".Net Project Builds with Node Package Manager" />
<published>2020-12-10T07:00:00+00:00</published>
<updated>2020-12-10T07:00:00+00:00</updated>
<id>https://lostechies.com/derekgreer/2020/12/10/dotnet-project-builds-with-npm</id>
<content type="html" xml:base="https://lostechies.com/derekgreer/2020/12/10/dotnet-project-builds-with-npm/">A few years ago, I wrote an article entitled <a href="http://aspiringcraftsman.com/2016/02/28/separation-of-concerns-application-builds-continuous-integration/">Separation of Concerns: Application Builds &amp; Continuous Integration</a> wherein I discussed the benefits of separating project builds from CI/CD concerns by creating a local build script which lives with your project. Not long after writing that article, I was turned on to what I’ve come to believe is one of the easiest tools I’ve encountered for managing .Net project builds thus far: npm.
Most development platforms provide a native task-based build technology. Microsoft’s tooling for these needs is MSBuild: a command-line tool whose build files double as Visual Studio’s project and solution definition files. I used MSBuild briefly for scripting custom build concerns for a couple of years, but found it to be awkward and cumbersome. Around 2007, I abandoned use of MSBuild for creating builds and began using Rake. While it had the downside of requiring a bit of knowledge of Ruby, it was a popular choice among those willing to look outside of the Microsoft camp for tooling and had community support for working with .Net builds through the <a href="https://www.codemag.com/article/1006101/Building-.NET-Systems-with-Ruby-Rake-and-Albacore">Albacore</a> library. I’ve used a few different technologies since, but about 5 years ago I saw a demonstration of the use of npm for building .Net projects at a conference and I was immediately sold. When used well, it really is the easiest and most terse way to script a custom build for the .Net platform I’ve encountered.
“So what’s special about npm?” you might ask. The primary appeal of using npm for building applications is that it’s easy to use. Essentially, it’s just an orchestration of shell commands.
<h3 id="tasks">Tasks</h3>
With other build tools, you’re often required to know a specific language in addition to learning special constructs peculiar to the build tool to create build tasks. In contrast, npm’s expected package.json file simply defines an array of shell command scripts:
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"name": "example",
"version": "1.0.0",
"description": "",
"scripts": {
"clean": "echo Clean the project.",
"restore": "echo Restore dependencies.",
"compile": "echo Compile the project.",
"test": "echo Run the tests.",
"dist": "echo Create a distribution."
},
"author": "Some author",
"license": "ISC"
}
</code></pre></div></div>
As with other build tools, NPM provides the ability to define dependencies between build tasks. This is done using pre- and post- lifecycle scripts. Simply, any task issued by NPM will first execute a script by the same name with a prefix of “pre” when present and will subsequently execute a script by the same name with a prefix of “post” when present. For example:
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"name": "example",
"version": "1.0.0",
"description": "",
"scripts": {
"clean": "echo Clean the project.",
"prerestore": "npm run clean",
"restore": "echo Restore dependencies.",
"precompile": "npm run restore",
"compile": "echo Compile the project.",
"pretest": "npm run compile",
"test": "echo Run the tests.",
"prebuild": "npm run test",
"build": "echo Publish a distribution."
},
"author": "Some author",
"license": "ISC"
}
</code></pre></div></div>
Based on the above package.json file, issuing “npm run build” will result in running the tasks of clean, restore, compile, test, and build in that order by virtue of each declaring an appropriate dependency.
Given you’re okay with limiting a fully-specified dependency chain where a subset of the build can be initiated at any stage (e.g. running “npm run test” and triggering clean, restore, and compile first) , the above orchestration can be simplified by installing the npm-run-all node dependency and defining a single pre- lifetime script for the main build target:
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"name": "example",
"version": "1.0.0",
"description": "",
"scripts": {
"clean": "echo Clean the project.",
"restore": "echo Restore dependencies.",
"compile": "echo Compile the project.",
"test": "echo Run the tests.",
"prebuild": "npm-run-all clean restore compile test",
"build": "echo Publish a distribution."
},
"author": "John Doe",
"license": "ISC",
"devDependencies": {
"npm-run-all": "^4.1.5"
}
}
</code></pre></div></div>
In this example, issuing “npm run build” will result in the prebuild script executing npm-run-all with the parameters: clean, restore, compile and test which it will execute in the order listed.
<h3 id="variables">Variables</h3>
Aside from understanding how to utilize the pre- and post- lifecycle scripts to denote task dependencies, the only other thing you really need to know is how to work with variables.
Node’s npm command facilitates the definition of variables by command-line parameters as well as declaring package variables. When npm executes, each of the properties declared within the package.json are flattened and prefixed with “npm_package_”. For example, the standard “version” property can be used as part of a dotnet build to denote a project version by referencing ${npm_package_version}:
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"name": "example",
"version": "1.0.0",
"description": "",
"configuration": "Release",
"scripts": {
"build": "dotnet build ./src/*.sln /p:Version=${npm_package_version}"
},
"author": "John Doe",
"license": "ISC",
"devDependencies": {
"npm-run-all": "^4.1.5"
}
}
</code></pre></div></div>
Command-line parameters can also be passed to npm and are similarly prefixed with “npm_config_” with any dashes (“-”) replaced with underscores (“_”). For example, the previous version setting could be passed to dotnet.exe in the following version of package.json by issuing the below command:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npm run build --product-version=2.0.0
</code></pre></div></div>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"name": "example",
"version": "1.0.0",
"description": "",
"configuration": "Release",
"scripts": {
"build": "dotnet build ./src/*.sln /p:Version=${npm_config_product_version}"
},
"author": "John Doe",
"license": "ISC",
"devDependencies": {
"npm-run-all": "^4.1.5"
}
}
</code></pre></div></div>
(Note: the parameter –version is an npm parameter for printing the version of npm being executed and therefore can’t be used as a script parameter.)
The only other important thing to understand about the use of variables with npm is that the method of dereferencing is dependent upon the shell used. When using npm on Windows, the default shell is cmd.exe. If using the default shell on Windows, the version parameter would need to be deference as %npm_config_product_version%:
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"name": "example",
"version": "1.0.0",
"description": "",
"configuration": "Release",
"scripts": {
"build": "dotnet build ./src/*.sln /p:Version=%npm_config_product_version%"
},
"author": "John Doe",
"license": "ISC",
"devDependencies": {
"npm-run-all": "^4.1.5"
}
}
</code></pre></div></div>
Until recently, I used a node package named “cross-env” which allows you to normalize how you dereference variables regardless of platform, but for several reasons including cross-env being placed in maintenance mode, the added dependency overhead, syntax noise, and support for advanced variable expansion cases such as default values, I’d recommend any cross-platform execution be supported by just standardizing on a single shell (e.g. “Bash”). With the introduction of Windows Subsystem for Linux and the virtual ubiquity of git for version control, most developer Windows systems already contain the bash shell. To configure npm to use bash at the project level, just create a file named .npmrc at the package root containing the following line:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>script-shell=bash
</code></pre></div></div>
<h3 id="using-node-packages">Using Node Packages</h3>
While not necessary, there are many CLI node packages that can be easily leveraged for aiding in authoring your builds. For example, a package named “rimraf”, which functions like Linux’s “rm -rf” command, is a utility you can use to implement a clean script for recursively deleting any temporary build folders created as part of previous builds. In the following package.json build, a package target builds a NuGet package which it outputs to a dist folder in the package root. The rimraf command is used to delete this temp folder as part of the build script’s dependencies:
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"name": "example",
"version": "1.0.0",
"description": "",
"scripts": {
"clean": "rimraf dist",
"prebuild": "npm run clean",
"build": "dotnet pack ./src/ExampleLibrary/ExampleLibrary.csproj -o dist /p:Version=${npm_package_version}"
},
"author": "John Doe",
"license": "ISC",
"devDependencies": {
"npm-run-all": "^4.1.5",
"rimraf": "^3.0.2"
}
}
</code></pre></div></div>
If you’d like to see a more complete example of npm at work, you can check out the <a href="https://github.com/derekgreer/conventional-options/blob/master/package.json">build for ConventionalOptions</a> which supports tasks for building, testing, packaging, and publishing nuget packages for both release and prerelease versions of the library.</content>
<author>
<name>Derek Greer</name>
</author>
<summary type="html">A few years ago, I wrote an article entitled Separation of Concerns: Application Builds &amp; Continuous Integration wherein I discussed the benefits of separating project builds from CI/CD concerns by creating a local build script which lives with your project. Not long after writing that article, I was turned on to what I’ve come to believe is one of the easiest tools I’ve encountered for managing .Net project builds thus far: npm.</summary>
</entry>
<entry>
<title type="html">Conventional Options</title>
<link href="https://lostechies.com/derekgreer/2020/11/20/conventional-options/" rel="alternate" type="text/html" title="Conventional Options" />
<published>2020-11-20T07:00:00+00:00</published>
<updated>2020-11-20T07:00:00+00:00</updated>
<id>https://lostechies.com/derekgreer/2020/11/20/conventional-options</id>
<content type="html" xml:base="https://lostechies.com/derekgreer/2020/11/20/conventional-options/">I’ve really enjoyed working with the Microsoft Configuration libraries introduced with .Net Core approximately 5 years ago. The older XML-based API was quite a pain to work with, so the ConfigurationBuilder and associated types provided a long overdue need for the platform.
I had long since adopted a practice of creating discrete configuration classes populated and registered with a DI container over direct use of the ConfigurationManager class within components, so I was pleased to see the platform nudge developers in this direction through the introduction of the IOptions<T> type.</T>
A few aspects surrounded the prescribed use of the IOptions<T> type of which I wasn't particularly fond were needing to inject IOptions<T> rather than the actual options type, taking a dependency upon the Microsoft.Extensions.Options package from my library packages, and the cermony of binding the options to the IConfiguration instance. To address these concerns, I wrote some extension methods which took care of binding the type to my configuration by convention (i.e. binding a type with a suffix of Options to a section corresponding to the option type's prefix) and registering it with the container.</T></T>
I’ve recently released a new version of these extensions supporting several of the most popular containers as an open source library. You can find the project <a href="http://github.com/derekgreer/conventional-options">here</a>.
The following are the steps for using these extensions:
<h3 id="step-1">Step 1</h3>
Install ConventionalOptions for the target DI container:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$&gt; nuget install ConventionalOptions.DependencyInjection
</code></pre></div></div>
<h3 id="step-2">Step 2</h3>
Add Microsoft’s Options feature and register option types:
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> services.AddOptions();
services.RegisterOptionsFromAssemblies(Configuration, Assembly.GetExecutingAssembly());
</code></pre></div></div>
<h3 id="step-3">Step 3</h3>
Create an Options class with the desired properties:
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public class OrderServiceOptions
{
public string StringProperty { get; set; }
public int IntProperty { get; set; }
}
</code></pre></div></div>
<h3 id="step-4">Step 4</h3>
Provide a corresponding configuration section matching the prefix of the Options class (e.g. in appsettings.json):
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"OrderService": {
"StringProperty": "Some value",
"IntProperty": 42
}
}
</code></pre></div></div>
<h3 id="step-5">Step 5</h3>
Inject the options into types resolved from the container:
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public class OrderService
{
public OrderService(OrderServiceOptions options)
{
// ... use options
}
}
</code></pre></div></div>
Currently ConventionalOptions works with Microsoft’s DI Container, Autofac, Lamar, Ninject, and StructureMap.
Enjoy!</content>
<author>
<name>Derek Greer</name>
</author>
<summary type="html">I’ve really enjoyed working with the Microsoft Configuration libraries introduced with .Net Core approximately 5 years ago. The older XML-based API was quite a pain to work with, so the ConfigurationBuilder and associated types provided a long overdue need for the platform.</summary>
</entry>
<entry>
<title type="html">Picking a Web Microframework</title>
<link href="https://lostechies.com/ryansvihla/2020/05/27/picking-a-microframework/" rel="alternate" type="text/html" title="Picking a Web Microframework" />
<published>2020-05-27T00:23:00+00:00</published>
<updated>2020-05-27T00:23:00+00:00</updated>
<id>https://lostechies.com/ryansvihla/2020/05/27/picking-a-microframework</id>
<content type="html" xml:base="https://lostechies.com/ryansvihla/2020/05/27/picking-a-microframework/">We had a “home grown” framework for a new application we’re working on and the first thing I did was try and rip that out (new project so didn’t have URL and parameter sanitization
anyway to do routes, etc).
However, being that the group I was working with has had some bad experiences with “frameworks” I had to settle on something that was lightweight, integrated
with Jetty and allowed us to work the way that was comfortable for us as team (also it had to work with Scala).
<h2 id="microframeworks">Microframeworks</h2>
The team had shown a lot of disdain for Play (which I had actually quite a lot when I last was leading a JVM based tech stack) and Spring Boot as being too heavy weight, so these
were definitely out.
Fortunately, in the JVM world there is a big push back now on heavy web frameworks so meant I had lots of choices for “non frameworks” but could
still do some basic security, routing, authentication but not hurt the existing team’s productivity.
There are probably 3 dozen microframeworks to choose from with varying degrees of value but the two that seemed to easiest to start with today were:
<ul>
<li><a href="https://scalatra.org">Scalatra</a></li>
<li><a href="https://javalin.io">Javalin</a></li>
<li><a href="https://quarkus.io">Quarkus</a></li>
</ul>
<h3 id="my-attempt-with-quarkus">My Attempt with Quarkus</h3>
<a href="https://quarkus.io/">Quarkus</a> has a really great getting started story but it’s harder to get started on an existing project with it, it was super trivial to add, and after a couple of days of figuring out the magic incantation I just decided to punt on it.
I think because of it’s popularity in the Cloud Native space (which we’re trying to target), the backing of <a href="https://developers.redhat.com/blog/2019/03/07/quarkus-next-generation-kubernetes-native-java-framework/">Red Hat</a>, and the pluggable nature of the stack there are a lot of reasons to want this to work.
In the end because of the timeline it didn’t make the cut. But it may come back.
<h3 id="my-attempt-with-javalin">My Attempt with Javalin</h3>
Javalin despite being a less popular project than Quarkus it is getting some buzz. It also looks like it just slides into the team’s existing Servlet code base. I wanted this to work very badly but stopped before
I even started because of <a href="https://github.com/tipsy/javalin/issues/931">this issue</a> so this was out despite being on paper a really execellent framework.
<h3 id="my-attempt-with-scalatra">My Attempt with Scalatra</h3>
<a href="https://scalatra.org/">Scalatra</a> has been around for a number of years and is inspired by <a href="http://sinatrarb.com/">Sinatra</a> which I used quite a bit in my Ruby years.
This took a few minutes to get going just following their <a href="https://scalatra.org/guides/2.7/deployment/standalone.html">standalone directions</a> and then some more to successful convert the routes
and account for learning curves with routes.
Some notes:
<ul>
<li>The routing API and parameters etc are very nice to work with IMO.</li>
<li>It was <a href="https://scalatra.org/guides/2.7/formats/json.html">very easy</a> to get json by default support setup.</li>
<li>Metrics were <a href="https://scalatra.org/guides/2.7/monitoring/metrics.html">very easy</a> to wire up.</li>
<li>Swagger integration was pretty rough, while it looks good on paper I could not get an example to show up, and it is unable to <a href="https://github.com/scalatra/scalatra/issues/343">handle case classes or enums</a> which we use.</li>
<li>Benchmark performance when I’ve <a href="https://johnykov.github.io/bootzooka-akka-http-vs-scalatra.html">looked</a> around the web was pretty bad, I’ve not done enough to figure out if this is real or not. I’ve seen first hand a lot of benchmarking are just wrong.</li>
<li>Integration with JUnit has been rough and I cannot seem to get the correct port to fire, I suspect I have to stop using the @Test annotation is all (which I’m not enjoying).</li>
<li>Http/2 support is still lacking despite being available in the version of Jetty they’re on, I’ve read a few places that an issue is keeping <a href="https://github.com/eclipse/jetty.project/issues/1364">web sockets working</a> but either way there is <a href="https://github.com/scalatra/scalatra/issues/757">no official support in the project yet</a>.</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
I think we’re going to stick with Scalatra for the time being as it is a muture framework that works well for our current goals. However, the lack of http/2 support maybe a deal breaker in the medium term.</content>
<author>
<name>Ryan Svihla</name>
</author>
<summary type="html">We had a “home grown” framework for a new application we’re working on and the first thing I did was try and rip that out (new project so didn’t have URL and parameter sanitization anyway to do routes, etc).</summary>
</entry>
<entry>
<title type="html">Getting started with Cassandra: Data modeling in the brief</title>
<link href="https://lostechies.com/ryansvihla/2020/02/05/getting-started-cassandra-part-3/" rel="alternate" type="text/html" title="Getting started with Cassandra: Data modeling in the brief" />
<published>2020-02-05T20:23:00+00:00</published>
<updated>2020-02-05T20:23:00+00:00</updated>
<id>https://lostechies.com/ryansvihla/2020/02/05/getting-started-cassandra-part-3</id>
<content type="html" xml:base="https://lostechies.com/ryansvihla/2020/02/05/getting-started-cassandra-part-3/">Cassandra data modeling isn’t really something you can do “in the brief” and is itself a subject that can take years to fully grasp, but this should be a good starting point.
<h2 id="introduction">Introduction</h2>
Cassandra distributes data around the cluster via the partition key.
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CREATE TABLE my_key.my_table_by_postal_code (postal_code text, id uuid, balance float, PRIMARY KEY(postal_code, id));
</code></pre></div></div>
In the above table the partition key is <code class="language-plaintext highlighter-rouge">postal_code</code> and the clustering column is<code class="language-plaintext highlighter-rouge">id</code>. The partition key will locate the data on the cluster for us. The clustering column allows us multiple rows per partition key so that we can filter how much data we read per partition.
The ‘optimal’ query is one that retrieves data from only one node and not so much data that GC pressure or latency issues result. The following query is breaking that rule and retrieving 2 partitions at once via the IN parameter.
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT * FROM my_key.my_table_by_postal_code WHERE postal_code IN ('77002', '77043');
</code></pre></div></div>
This can be slower than doing two separate queries asynchronously, especially if those partitions are on two different nodes (imagine if there are 1000+ partitions in the IN statement). In summary, the simple rule to stick to is “1 partition per query”.
<h3 id="partition-sizes">Partition sizes</h3>
A common mistake when data modeling is to jam as much data as possible into a single partition.
<ul>
<li>This doesn’t distribute the data well and therefore misses the point of a distributed database.</li>
<li>There are practical limits on the <a href="https://issues.apache.org/jira/browse/CASSANDRA-9754">performance of partition sizes</a></li>
</ul>
<h3 id="table-per-query-pattern">Table per query pattern</h3>
A common approach to optimize around partition lookup is to create a table per query, and write to all of them on update. The following example has two related tables both to solve two different queries
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>--query by postal_code
CREATE TABLE my_key.my_table_by_postal_code (postal_code text, id uuid, balance float, PRIMARY KEY(postal_code, id));
SELECT * FROM my_key.my_table_by_postal_code WHERE postal_code = '77002';
--query by id
CREATE TABLE my_key.my_table (id uuid, name text, address text, city text, state text, postal_code text, country text, balance float, PRIMARY KEY(id));
SELECT * FROM my_key.my_table WHERE id = 7895c6ff-008b-4e4c-b0ff-ba4e4e099326;
</code></pre></div></div>
You can update both tables at once with a logged batch:
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>BEGIN BATCH
INSERT INTO my_key.my_table (id, name, address, city, state, postal_code, country, balance) VALUES (7895c6ff-008b-4e4c-b0ff-ba4e4e099326, 'Bordeaux', 'Gironde', '33000', 'France', 56.20);
INSERT INTO my_key.my_table_by_postal_code (postal_code, id, balance) VALUES ('33000', 7895c6ff-008b-4e4c-b0ff-ba4e4e099326, 56.20) ;
APPLY BATCH;
</code></pre></div></div>
<h3 id="source-of-truth">Source of truth</h3>
A common design pattern is to have one table act as the authoritative one over data, and if for some reason there is a mismatch or conflict in other tables as long as there is one considered “the source of truth” it makes it easy to fix any conflicts later. This is typically the table that would match what we see in typical relational databases and has all the data needed to generate all related views or indexes for different query methods. Taking the prior example, <code class="language-plaintext highlighter-rouge">my_table</code> is the source of truth:
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>--source of truth table
CREATE TABLE my_key.my_table (id uuid, name text, address text, city text, state text, postal_code text, country text, balance float, PRIMARY KEY(id));
SELECT * FROM my_key.my_table WHERE id = 7895c6ff-008b-4e4c-b0ff-ba4e4e099326;
--based on my_key.my_table and so we can query by postal_code
CREATE TABLE my_key.my_table_by_postal_code (postal_code text, id uuid, balance float, PRIMARY KEY(postal_code, id));
SELECT * FROM my_key.my_table_by_postal_code WHERE postal_code = '77002';
</code></pre></div></div>
Next we discuss strategies for keeping tables of related in sync.
<h3 id="materialized-views">Materialized views</h3>
Materialized views are a feature that ships with Cassandra but is currently considered rather experimental. If you want to use them anyway:
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CREATE MATERIALIZED VIEW my_key.my_table_by_postal_code
AS SELECT postal_code text, id uuid, balance float
FROM my_key.my_table
WHERE postal_code IS NOT NULL AND id IS NOT NULL
PRIMARY KEY(postal_code, id));
</code></pre></div></div>
Materialized views at least run faster than the comparable BATCH insert pattern, but they have a number of bugs and known issues that are still pending fixes.
<h3 id="secondary-indexes">Secondary indexes</h3>
This are the original server side approach to handling different query patterns but it has a large number of downsides:
<ul>
<li>rows are read serially one node at time until limit is reached.</li>
<li>a suboptimal storage layout leading to very large partitions if the data distribution of the secondary index is not ideal.</li>
</ul>
For just those two reasons I think it’s rare that one can use secondary indexes and expect reasonable performance. However, you can make one by hand and just query that data asynchronously to avoid some of the downsides.
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CREATE TABLE my_key.my_table_by_postal_code_2i (postal_code text, id uuid, PRIMARY KEY(postal_code, id));
SELECT * FROM my_key.my_table_by_postal_code_2i WHERE postal_code = '77002';
--retrieve all rows then asynchronously query the resulting ids
SELECT * FROM my_key.my_table WHERE id = ad004ff2-e5cb-4245-94b8-d6acbc22920a;
SELECT * FROM my_key.my_table WHERE id = d30e9c65-17a1-44da-bae0-b7bb742eefd6;
SELECT * FROM my_key.my_table WHERE id = e016ae43-3d4e-4093-b745-8583627eb1fe;
</code></pre></div></div>
<h2 id="exercises">Exercises</h2>
<h3 id="contact-list">Contact List</h3>
This is a good basic first use case as one needs to use multiple tables for the same data, but there should not be too many.
<h4 id="requirements">requirements</h4>
<ul>
<li>contacts should have first name, last name, address, state/region, country, postal code</li>
<li>lookup by contacts id</li>
<li>retrieve all contacts by a given last name</li>
<li>retrieve counts by zip code</li>
</ul>
<h3 id="music-service">Music Service</h3>
Takes the basics from the previous exercise and requires a more involved understanding of the concepts. It will require many tables and some difficult trade-offs on partition sizing. There is no one correct way to do this.
<h4 id="requirements-1">requirements</h4>
<ul>
<li>songs should have album, artist, name, and total likes</li>
<li>The contact list exercise, can be used as a basis for the “users”, users will have no login because we’re trusting people</li>
<li>retrieve all songs by artist</li>
<li>retrieve all songs in an album</li>
<li>retrieve individual song and how many times it’s been liked</li>
<li>retrieve all liked songs for a given user</li>
<li>“like” a song</li>
<li>keep a count of how many times a song has been listened to by all users</li>
</ul>
<h3 id="iot-analytics">IoT Analytics</h3>
This will require some extensive time series modeling and takes some of the lessons from the Music Service further. The table(s) used will be informed by the query.
<h4 id="requirements-2">requirements</h4>
<ul>
<li>use the music service data model as a basis, we will be tracking each “registered device” that uses the music service</li>
<li>a given user will have 1-5 devices</li>
<li>log all songs listened to by a given device</li>
<li>retrieve songs listened for a device by day</li>
<li>retrieve songs listened for a device by month</li>
<li>retrieve total listen time for a device by day</li>
<li>retrieve total listen time for a device by month</li>
<li>retrieve artists listened for a device by day</li>
<li>retrieve artists listened for a device by month</li>
</ul></content>
<author>
<name>Ryan Svihla</name>
</author>
<summary type="html">Cassandra data modeling isn’t really something you can do “in the brief” and is itself a subject that can take years to fully grasp, but this should be a good starting point.</summary>
</entry>
<entry>
<title type="html">Getting started with Cassandra: Load testing Cassandra in brief</title>
<link href="https://lostechies.com/ryansvihla/2020/02/04/getting-started-cassandra-part-2/" rel="alternate" type="text/html" title="Getting started with Cassandra: Load testing Cassandra in brief" />
<published>2020-02-04T20:23:00+00:00</published>
<updated>2020-02-04T20:23:00+00:00</updated>
<id>https://lostechies.com/ryansvihla/2020/02/04/getting-started-cassandra-part-2</id>
<content type="html" xml:base="https://lostechies.com/ryansvihla/2020/02/04/getting-started-cassandra-part-2/">An opinionated guide on the “correct” way to load test Cassandra. I’m aiming to keep this short so I’m going to leave out a lot of the nuance that one would normally get into when talking about load testing cassandra.
<h2 id="if-you-have-no-data-model-in-mind">If you have no data model in mind</h2>
Use cassandra stress since it’s around:
<ul>
<li>first initialize the keyspace with RF3 <code class="language-plaintext highlighter-rouge">cassandra-stress "write cl=ONE no-warmup -col size=FIXED(15000) -schema replication(strategy=SimpleStrategy,factor=3)"</code></li>
<li>second run stress <code class="language-plaintext highlighter-rouge">cassandra-stress "mixed n=1000k cl=ONE -col size=FIXED(15000)</code></li>
<li>repeat as often as you’d like with as many clients as you want.</li>
</ul>
<h2 id="if-you-have-a-specific-data-model-in-mind">If you have a specific data model in mind</h2>
You can use cassandra-stress, but I suspect you’re going to find your data model isn’t supported (collections for example) or that you don’t have the required PHD to make it work the way you want. There are probably 2 dozen options from here you can use to build your load test, some of the more popular ones are gatling, jmeter, and tlp-stress. My personal favorite for this though, write a small simple python or java program that replicates your use case accurately in your own code, using a faker library to generate your data. This takes more time but you tend to have less surprises in production as it will accurately model your code.
<h3 id="small-python-script-with-python-driver">Small python script with python driver</h3>
<ul>
<li>use python3 and virtualenv</li>
<li><code class="language-plaintext highlighter-rouge">python -m venv venv</code></li>
<li>source venv/bin/activate</li>
<li>read and follow install <a href="https://docs.datastax.com/en/developer/python-driver/3.21/getting_started/">docs</a></li>
<li>if you want to skip the docs you can get away with <code class="language-plaintext highlighter-rouge">pip install cassandra-driver</code></li>
<li>install a faker library <code class="language-plaintext highlighter-rouge">pip install Faker</code></li>
</ul>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import argparse
import uuid
import time
import random
from cassandra.cluster import Cluster
from cassandra.query import BatchStatement
from faker import Faker
parser = argparse.ArgumentParser(description='simple load generator for cassandra')
parser.add_argument('--hosts', default='127.0.0.1',
type=str,
help='comma separated list of hosts to use for contact points')
parser.add_argument('--port', default=9042, type=int, help='port to connect to')
parser.add_argument('--trans', default=1000000, type=int, help='number of transactions')
parser.add_argument('--inflight', default=25, type=int, help='number of operations in flight')
parser.add_argument('--errors', default=-1, type=int, help='number of errors before stopping. default is unlimited')
args = parser.parse_args()
fake = Faker(['en-US'])
hosts = args.hosts.split(",")
cluster = Cluster(hosts, port=args.port)
try:
session = cluster.connect()
print("setup schema");
session.execute("CREATE KEYSPACE IF NOT EXISTS my_key WITH REPLICATION = {'class': 'SimpleStrategy', 'replication_factor': 1}")
session.execute("CREATE TABLE IF NOT EXISTS my_key.my_table (id uuid, name text, address text, state text, zip text, balance int, PRIMARY KEY(id))")
session.execute("CREATE TABLE IF NOT EXISTS my_key.my_table_by_zip (zip text, id uuid, balance bigint, PRIMARY KEY(zip, id))")
print("allow schema to replicate throughout the cluster for 30 seconds")
time.sleep(30)
print("prepare queries")
insert = session.prepare("INSERT INTO my_key.my_table (id, name, address, state, zip, balance) VALUES (?, ?, ?, ?, ?, ?)")
insert_rollup = session.prepare("INSERT INTO my_key.my_table_by_zip (zip, id, balance) VALUES (?, ?, ?)")
row_lookup = session.prepare("SELECT * FROM my_key.my_table WHERE id = ?")
rollup = session.prepare("SELECT sum(balance) FROM my_key.my_table_by_zip WHERE zip = ?")
threads = []
ids = []
error_counter = 0
query = None
params = []
ids = []
def get_id():
items = len(ids)
if items == 0:
## nothing present so return something random
 return uuid.uuid4()
if items == 1:
return ids[0]
return ids[random.randint(0, items -1)]
print("starting transactions")
for i in range(args.trans):
chance = random.randint(1, 100)
if chance &gt; 0 and chance &lt; 50:
new_id = uuid.uuid4()
ids.append(new_id)
state = fake.state_abbr()
zip_code = fake.zipcode_in_state(state)
balance = random.randint(1, 50000)
query = BatchStatement()
name = fake.name()
address = fake.address()
bound_insert = insert.bind([new_id, fake.name(), fake.address(), state, zip_code, balance])
query.add(bound_insert)
bound_insert_rollup = insert_rollup.bind([zip_code, new_id, balance])
query.add(bound_insert_rollup)
elif chance &gt; 50 and chance &lt; 75:
query = row_lookup.bind([get_id()])
elif chance &gt; 75:
zip_code = fake.zipcode()
query = rollup.bind([zip_code])
threads.append(session.execute_async(query))
if i % args.inflight == 0:
for t in threads:
try:
t.result() #we don't care about result so toss it
 except Exception as e:
print("unexpected exception %s" % e)
if args.errors &gt; 0:
error_counter = error_counter + 1
if error_counter &gt; args.errors:
print("too many errors stopping. Consider raising --errors flag if this happens more quickly than you'd like")
break
threads = []
print("submitted %i of %i transactions" % (i, args.trans))
finally:
cluster.shutdown()
</code></pre></div></div>
<h3 id="small-java-program-with-latest-java-driver">Small java program with latest java driver</h3>
<ul>
<li>download java 8</li>
<li>create a command line application in your project technology of choice (I used maven in this example for no particularly good reason)</li>
<li>download a faker lib like <a href="https://github.com/DiUS/java-faker">this one</a> and the <a href="https://github.com/datastax/java-driver">Cassandra java driver from DataStax</a> again using your preferred technology to do so.</li>
<li>run the following code sample somewhere (set your RF and your desired queries and data model)</li>
<li>use different numbers of clients at your cluster until you get enough “saturation” or the server stops responding.</li>
</ul>
<a href="https://github.com/rssvihla/simple_cassandra_load_test/tree/master/java/simple-cassandra-stress">See complete example</a>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code>package pro.foundev;
import java.lang.RuntimeException;
import java.lang.Thread;
import java.util.Locale;
import java.util.ArrayList;
import java.util.List;
import java.util.function.*;
import java.util.Random;
import java.util.UUID;
import java.util.concurrent.CompletionStage;
import java.net.InetSocketAddress;
import com.datastax.oss.driver.api.core.CqlSession;
import com.datastax.oss.driver.api.core.CqlSessionBuilder;
import com.datastax.oss.driver.api.core.cql.*;
import com.github.javafaker.Faker;
public class App
{
public static void main( String[] args )
{
List&lt;String&gt; hosts = new ArrayList&lt;&gt;();
hosts.add("127.0.0.1");
if (args.length &gt; 0){
hosts = new ArrayList&lt;&gt;();
String rawHosts = args[0];
for (String host: rawHosts.split(",")){
hosts.add(host.trim());
}
}
int port = 9042;
if (args.length &gt; 1){
port = Integer.valueOf(args[1]);
}
long trans = 1000000;
if (args.length &gt; 2){
trans = Long.valueOf(args[2]);
}
int inFlight = 25;
if (args.length &gt; 3){
inFlight = Integer.valueOf(args[3]);
}
int maxErrors = -1;
if (args.length &gt; 4){
maxErrors = Integer.valueOf(args[4]);
}
CqlSessionBuilder builder = CqlSession.builder();
for (String host: hosts){
builder = builder.addContactPoint(new InetSocketAddress(host, port));
}
builder = builder.withLocalDatacenter("datacenter1");
try(final CqlSession session = builder.build()){
System.out.println("setup schema");
session.execute("CREATE KEYSPACE IF NOT EXISTS my_key WITH REPLICATION = {'class': 'SimpleStrategy', 'replication_factor': 1}");
session.execute("CREATE TABLE IF NOT EXISTS my_key.my_table (id uuid, name text, address text, state text, zip text, balance int, PRIMARY KEY(id))");
session.execute("CREATE TABLE IF NOT EXISTS my_key.my_table_by_zip (zip text, id uuid, balance bigint, PRIMARY KEY(zip, id))");
System.out.println("allow schema to replicate throughout the cluster for 30 seconds");
try{
Thread.sleep(30000);
}catch(Exception ex){
throw new RuntimeException(ex);
}
System.out.println("prepare queries");
final PreparedStatement insert = session.prepare("INSERT INTO my_key.my_table (id, name, address, state, zip, balance) VALUES (?, ?, ?, ?, ?, ?)");
final PreparedStatement insertRollup = session.prepare("INSERT INTO my_key.my_table_by_zip (zip, id, balance) VALUES (?, ?, ?)");
final PreparedStatement rowLookup = session.prepare("SELECT * FROM my_key.my_table WHERE id = ?");
final PreparedStatement rollup = session.prepare("SELECT sum(balance) FROM my_key.my_table_by_zip WHERE zip = ?");
final List&lt;UUID&gt; ids = new ArrayList&lt;&gt;();
final Random rnd = new Random();
final Locale us = new Locale("en-US");
final Faker faker = new Faker(us);
final Supplier&lt;UUID&gt; getId = ()-&gt; {
if (ids.size() == 0){
//return random uuid will be record not found
return UUID.randomUUID();
}
if (ids.size() == 1){
return ids.get(0);
}
final int itemIndex = rnd.nextInt(ids.size()-1);
return ids.get(itemIndex);
};
final Supplier&lt;Statement&lt;?&gt;&gt; getOp = ()-&gt; {
int chance = rnd.nextInt(100);
if (chance &gt; 0 &amp;&amp; chance &lt; 50){
final String state = faker.address().stateAbbr();
final String zip = faker.address().zipCodeByState(state);
final UUID newId = UUID.randomUUID();
final int balance = rnd.nextInt();
ids.add(newId);
return BatchStatement.builder(BatchType.LOGGED)
.addStatement(insert.bind(newId,
faker.name().fullName(),
faker.address().streetAddress(),
state,
zip,
balance))
.addStatement(insertRollup.bind(zip, newId, Long.valueOf(balance)))
.build();
} else if (chance &gt; 50 &amp;&amp; chance &lt; 75){
return rowLookup.bind(getId.get());
}
final String state = faker.address().stateAbbr();
final String zip = faker.address().zipCodeByState(state);
return rollup.bind(zip);
};
System.out.println("start transactions");
List&lt;CompletionStage&lt;AsyncResultSet&gt;&gt; futures = new ArrayList&lt;&gt;();
int errorCounter = 0;
for (int i = 0; i &lt; trans; i++){
//this is an uncessary hack to port old code and cap transactions in flight
if ( i % inFlight == 0){
for (CompletionStage&lt;AsyncResultSet&gt; future: futures){
try{
future.thenRun(()-&gt;{});
}catch(Exception ex){
if (maxErrors &gt; 0){
if (errorCounter &gt; maxErrors){
System.out.println("too many errors therefore stopping.");
break;
}
errorCounter += 1;
}
}
}
futures = new ArrayList&lt;&gt;();
System.out.println("submitted " + Integer.toString(i) + " of " + Long.toString(trans) + " transactions");
}
Statement&lt;?&gt; query = getOp.get();
futures.add(session.executeAsync(query));
}
}
}
}
</code></pre></div></div>
<h2 id="how-to-measure-performance">How to measure performance</h2>
The above scripts and examples are not ideal for several points but if you throw enough clients at the cluster at the same time those problems should balance out (namely the pseudo random distributions aren’t ideal).
But let’s assume you’ve worked these points out or gone with a gatling or other such tool what sort of issues should you look for now:
<ul>
<li>Are the nodes saturated or not? IE have you thrown enough client load at them?</li>
<li>What does GC look like? Use a tool like <a href="gceasy.io">gceasy</a> with the logs or <a href="https://www.github.com/riptano/sperf">sperf core gc</a> to analyze things.</li>
<li>How are pending compactions and pending mutations looking? Again, a tool like <a href="https://www.github.com/riptano/sperf">sperf core statuslogger</a> help a lot. Rule of thumb though is more than 100 pending compactions or 10000 pending mutations spells trouble.</li>
<li>How does disk io and cpu usage look? Collect an iostat on your servers. Disk queue length of 1 means your IO isn’t keeping up, this will happen sometimes in a busy but otherwise healthy server, if it’s happening a lot (more than 5%) of the time you’re in for some trouble. If your CPU uses hyperthreading and is busier than 40% of the time a lot of the time (say more than 5% again) that’s probably alos an issue. Use <code class="language-plaintext highlighter-rouge">sperf sysbottle</code> to analyze your iostat file.</li>
</ul>
<h2 id="summary">Summary</h2>
There are dozens of other good measurements to look at when trying to monitor load on your server and observe what your server can handle, but this is supposed to be a quick and dirty guide. So try these out for starters and ask yourself the following questions:
<ul>
<li>Does my data model generate a lot of GC? If so how can I change it? Validate this by randomly turniing off some of your queries and seeing which ones are the most expensive.</li>
<li>Is my server well tuned for my hardware? Is CPU or IO pegged? If not how busy are they? Can you add more clients? If not why?</li>
<li>Should I just add more nodes? Does changing RF have any affect on my reads and write load?</li>
</ul></content>
<author>
<name>Ryan Svihla</name>
</author>
<summary type="html">An opinionated guide on the “correct” way to load test Cassandra. I’m aiming to keep this short so I’m going to leave out a lot of the nuance that one would normally get into when talking about load testing cassandra.</summary>
</entry>
<entry>
<title type="html">Getting started with Cassandra: Setting up a Multi-DC environment</title>
<link href="https://lostechies.com/ryansvihla/2020/02/03/getting-started-cassandra-part-1/" rel="alternate" type="text/html" title="Getting started with Cassandra: Setting up a Multi-DC environment" />
<published>2020-02-03T20:23:00+00:00</published>
<updated>2020-02-03T20:23:00+00:00</updated>
<id>https://lostechies.com/ryansvihla/2020/02/03/getting-started-cassandra-part-1</id>
<content type="html" xml:base="https://lostechies.com/ryansvihla/2020/02/03/getting-started-cassandra-part-1/">This is a quick and dirty opinionated guide to setting up a Cassandra cluster with multiple data centers.
<h2 id="a-new-cluster">A new cluster</h2>
<ul>
<li>In cassandra.yaml set <code class="language-plaintext highlighter-rouge">endpoint_snitch: GossipingPropertyFileSnitch</code>, some prefer PropertyFileSnitch for the ease of pushing out one file. GossipingPropertyFileSnitch is harder to get wrong in my experience.</li>
<li>set dc in cassandra-rackdc.properties. Set to be whatever dc you want that node to be in. Ignore rack until you really need it, 8/10 people that use racks do it wrong the first time, and it’s slightly painful to unwind.</li>
<li>finish adding all of your nodes.</li>
<li>if using authentication, set <code class="language-plaintext highlighter-rouge">system_auth</code> keyspace to use NetworkTopologyStrategy in cqlsh with RF 3 (or == number of replicas if less than 3 per dc) for each datacenter you’ve created <code class="language-plaintext highlighter-rouge">ALTER KEYSPACE system_auth WITH REPLICATION= {'class' : 'NetworkTopologyStrategy', 'data_center_name' : 3, 'data_center_name' : 3};</code>, run repair after changing RF</li>
<li><code class="language-plaintext highlighter-rouge">nodetool repair -pr system_auth</code> on each node in the cluster on the new keyspace.</li>
<li>create your new keyspaces for your app with RF 3 in each dc (much like you did for the <code class="language-plaintext highlighter-rouge">system_auth</code> step above).</li>
<li><code class="language-plaintext highlighter-rouge">nodetool repair -pr whatever_new_keyspace</code> on each node in the cluster on the new keyspace.</li>
</ul>
<h2 id="an-existing-cluster">An existing cluster</h2>
This is harder and involves more work and more options, but I’m going to discuss the way that gets you into the least amount of trouble operationally.
<ul>
<li>make sure none of the drivers you use to connect to cassnadra are using DowngradingConsistencyRetryPolicy, or using the maligned withUsedHostsPerRemoteDc, especially allowRemoteDCsForLocalConsistencyLevel, as this may cause your driver to send requests to the remote data center before it’s populated with data.</li>
<li>switch <code class="language-plaintext highlighter-rouge">endpoint_snitch</code> on each node to GossipingPropertyFileSnitch</li>
<li>set dc in cassandra-rackdc.properties. Set to be whatever dc you want that node to be in. Ignore rack until you really need it, 8/10 people that use racks do it wrong the first time, and it’s slightly painful to unwind.</li>
<li>bootstrap each node in the new data center.</li>
<li>if using authentication, set <code class="language-plaintext highlighter-rouge">system_auth</code> keyspace to use NetworkTopologyStrategy in cqlsh with RF 3 (or == number of replicas if less than 3 per dc) for each datacenter you’ve created <code class="language-plaintext highlighter-rouge">ALTER KEYSPACE system_auth WITH REPLICATION= {'class' : 'NetworkTopologyStrategy', 'data_center_name' : 3, 'data_center_name' : 3};</code>, run repair after changing RF</li>
<li><code class="language-plaintext highlighter-rouge">nodetool repair -pr system_auth</code> on each node in the cluster on the new keyspace.</li>
<li>alter your app keyspaces for your app with RF 3 in each dc (much like you did for the <code class="language-plaintext highlighter-rouge">system_auth</code> step above),</li>
<li><code class="language-plaintext highlighter-rouge">nodetool repair -pr whatever_keyspace</code> on each node in the cluster on the new keyspace.</li>
</ul>
enjoy new data center
<h3 id="how-to-get-data-to-new-dc">how to get data to new dc</h3>
<h4 id="repair-approach">Repair approach</h4>
Best done with if your repair jobs can’t be missed or stopped, either because you have a process like opscenter or repear running repairs. It also has the advantage of being very easy, and if you’ve already automated repair you’re basically done.
<ul>
<li>let repair jobs continue..that’s it!</li>
</ul>
<h4 id="rebuild-approach">Rebuild approach</h4>
Faster less resource intensive, and if you have enough time to complete it while repair is stopped. Rebuild is easier to ‘resume’ than repair in many ways, so this has a number of advantages.
<ul>
<li>run <code class="language-plaintext highlighter-rouge">nodetool rebuild</code> on each node in the new dc only, if it dies for some reason, rerunning the command will resume the process.</li>
<li>run <code class="language-plaintext highlighter-rouge">nodetool cleanup</code></li>
</ul>
<h4 id="yolo-rebuild-with-repair">YOLO rebuild with repair</h4>
This will probably overstream it’s share of data and honestly a lot of folks do this for some reason in practice:
<ul>
<li>leave repair jobs running</li>
<li>run <code class="language-plaintext highlighter-rouge">nodetool rebuild</code> on each node in the new dc only, if it dies for some reason, rerunning the command will resume the process.</li>
<li>run <code class="language-plaintext highlighter-rouge">nodetool cleanup</code> on each node</li>
</ul>
<h2 id="cloud-strategies">Cloud strategies</h2>
There are a few valid approaches to this and none of them are wrong IMO.
<h3 id="region--dc-rack--az">region == DC, rack == AZ</h3>
Will need to get into racks and a lot of people get this wrong and imbalance the racks, but you get the advantage of more intelligent failure modes, with racks mapping to AZs.
<h3 id="azregardless-of-region--dc">AZ..regardless of region == DC</h3>
This allows things to be balanced easily, but you have no good option for racks then. However, some people think racks are overated, and I’d say a majority of clusters run with one rack.</content>
<author>
<name>Ryan Svihla</name>
</author>
<summary type="html">This is a quick and dirty opinionated guide to setting up a Cassandra cluster with multiple data centers.</summary>
</entry>
<entry>
<title type="html">MVP how minimal</title>
<link href="https://lostechies.com/ryansvihla/2018/12/20/mvp-how-minimal/" rel="alternate" type="text/html" title="MVP how minimal" />
<published>2018-12-20T20:00:00+00:00</published>
<updated>2018-12-20T20:00:00+00:00</updated>
<id>https://lostechies.com/ryansvihla/2018/12/20/mvp-how-minimal</id>
<content type="html" xml:base="https://lostechies.com/ryansvihla/2018/12/20/mvp-how-minimal/">MVPs or Minimum Viable Products are pretty contentious ideas for something seemingly simple. Depending on background and where pepole are coming from experience wise those terms carry radically different ideas.
In recent history I’ve seen up close two extreme constrasting examples of MVP:
<ul>
<li>Mega Minimal: website and db, mostly manual on the backend</li>
<li>Mega Mega: provisioning system, dynamic tuning of systems via ML, automated operations, monitoring a few others I’m leaving out.</li>
</ul>
<h2 id="feedback">Feedback</h2>
If we’re evaluating which approach gives us more feedback, Mega Minimal MVP is gonna win hands down here. Some will
counter they don’t want to give people a bad impression with a limited product and that’s fair, but it’s better than no impression (the dreaded never shipped MVP). The Mega Mega
MVP I referenced took months to demo. only had one of those checkboxes setup and wasn’t ever demod again. So we can categorical say that failed
at getting any feedback.
Whereas the Mega Minimal MVP, got enough feedback and users for the founders to realize that wasn’t a business for them.
Better than after hiring a huge team and sinking a million plus into dev efforts for sure. Not the happy ending I’m sure you all were expecting, but I view that as mission accomplished.
<h2 id="core-value">Core Value</h2>
<ul>
<li>Mega Minimal, they only focused on a single feature, executed well enough that people gave them some positive feedback, but not enough to justify automating everything.</li>
<li>Mega Mega. I’m not sure anyone who talked about the product saw the same core value, and there were several rewrites and shifts along the way.</li>
</ul>
Advantage Mega Minimal again
<h2 id="what-about-entrants-into-a-crowded-field">What about entrants into a crowded field</h2>
Well that is harder and the MVP tends to be less minimal, because the baseline expectations are just much higher. I still lean towards Mega Minimal having a better chance at getting users, since there is a non zero chance the Mega Mega MVP will never get finished.
I still think the exercise in focusing on core value that makes your product not a me too, and even considering how you can find a niche in a crowded field
instead of just being “better”, and your MVP can be that niche differentiator.
<h2 id="internal-users">Internal users</h2>
Sometimes a good middle ground is considering getting lots of internal users if you’re really worried about bad experiences. This has it’s
it’s definite downsides however, and you may not get diverse enough opinions. But it does give you some feedback while saving some face or bad experiences. I often think of the example of EC2 that was heavily used by Amazon, before being released to the world.
That was a luxury Amazon had, where their customer base and their user base happened to be very similar, and they had bigger scale needs than any of their early customers, so the early internal feedback loop was a very strong signal.
<h2 id="summary">Summary</h2>
In the end however you want to approach MVPs is up to you, and if you find success with a meatier MVP than I have please don’t let me push you away from what works. But if you are having trouble shipping and are getting pushed all the time to add one more feature to that MVP before releasing it, consider stepping back and asking is this really core value
for the product? Do you already have your core value? if so, consider just releasing it.</content>
<author>
<name>Ryan Svihla</name>
</author>
<summary type="html">MVPs or Minimum Viable Products are pretty contentious ideas for something seemingly simple. Depending on background and where pepole are coming from experience wise those terms carry radically different ideas. In recent history I’ve seen up close two extreme constrasting examples of MVP:</summary>
</entry>
</feed>

If you would like to create a banner that links to this page (i.e. this validation result), do the following:

Download the "valid Atom 1.0" banner.
Upload the image to your own server. (This step is important. Please do not link directly to the image on this server.)
Add this HTML to your page (change the image src attribute if necessary):

<a href="http://www.feedvalidator.org/check.cgi?url=https%3A//feeds.feedburner.com/LosTechies"><img src="valid-atom.png" alt="[Valid Atom 1.0]" title="Validate my Atom 1.0 feed" /></a>

If you would like to create a text link instead, here is the URL you can use:

http://www.feedvalidator.org/check.cgi?url=https%3A//feeds.feedburner.com/LosTechies

Home · About · News · Docs · Terms