automagical.rationalmind.netAuto-Magical

Search Primary Menu Skip to content About David Search for: tutorials FEE.org Project Retrospective Part 3: tools for managing software projects September 28, 2015 David Davidson 1 Comment This is the third part in my series on the migration of fee.org to Umbraco. This post will cover the tools and processes I used to manage this project. Development Process Overview Here is a high level overview of the implementation process – some of the the development steps were done in parallel: Interview stakeholders – understand their workflow and goals for the new site Perform in-depth site survey – build a sitemap to understand site structure Document types/page field survey – survey all fields (properties) used on each page, think about how they are related Template design – a combination of scraping the visual design of the old site and building static HTML for the new one Document type design – implement the document types in Umbraco based on the Page Field survey results Custom functionality – implement all the custom functionality not provided by Umbraco – payments, custom widgets, etc. Content migration – prepare a set of migration tools which would be run at the time of the switch Live release – execute the switch, including the final migration of content and media from old site to new. Tools Overview: Basecamp : basecamp was used in lieu of email for information discussions. JIRA : JIRA is used to manage all development tasks BitBucket (git): contains source code & database scripts, TeamCity : build server used for continous integration of the dev servers and production releases Amazon Web Services : hosts the application, include website, DB, email, storage, etc Hangouts /Skype: Permanent Hangout is used for ongoing discussion, weekly video calls on Hangout and Skype for team meetings. Evernote : a project notebook contains interview notes and other various technical snippets which I might need to refer to LastPass : password manager which I use to store all project credentials and share them with the team Project Tools in Detail: JIRA Release Schedule: A JIRA project is the first development artifact that I create. It contains the development plan, all the individual dev tasks, time tracking, and links to git commits. A project schedule is important to stakeholders, so I organize a high-level tasks list into pre-launch and post-launch releases: JIRA Kanban Board The JIRA Agile Board is how I organize my tasks. If there is a dedicated PM who is familiar with scrum, I will use the scrum board, otherwise I will use the more flexible Kanban board. I configured it into the standard four lanes: JIRA Task Detail: Pretty standard. I link JIRA to git so the commits for each task are visible, and also to TeamCity, so that the build status for each commit is linked. As I work on stories, I add screenshots and technical notes, for myself as much as the tester/product owner. BitBucket I use a modified git-flow proces – each commit is tagged with a build, releases are tagged by date, and released code is merged to master. TeamCity Each commit is automatically deployed to the dev site. Live releases are triggered from TeamCity took. I configure the Publish Web wizard in Visual Studio – this creates an msbuild configuration which I can trigger in the TeamCity build: New Relic Monitoring New Relic is pretty essential to running lots of websites without an ops team. It sends alerts when there is any problem and makes it easy to identity problematic components. Evernote Everynote contains technical notes, potential third party components, and client interviews, airplane tickets, and a lot of other information I may need to refer to: PowerPoint Last but not least, I write regular emails and reports to communicate project status information to non-technical stakeholders and educate them about the development process. tips Avoiding the Performance Panic Spiral of Doom September 28, 2015 David Davidson Leave a comment The following warning applies to anytime you try to fix a misbehaving system without understanding the cause of the problem, but especially relevant when trying to fix performance issues without knowing the cause: The trouble started when the site started randomly slowing to a crawl at random times. The tech team met to discuss the issue. Having failed to extract the cause by the act of stuffing enough smart people in a room, the topic shifted to solutions. “Let’s switch our caching from memcached to redis” I said. The testing went well, and the change was made. The following testing, accompanied by a dose optimistic thinking, let to the conclusion that the issue was improved. Everyone was happy, until it was discovered that the registration system was broken, because in one specific function, PHP failed to set the redis cache, causing a redirect loop. We fixed the problem, but the performance issue returned.Following this, another dozen configuration and code changes were tried. Since we could not consistently reproduce the performance issue, it was questionable whether any of these changes helped. The only thing which became clear was that our site was becoming increasingly unstable, and we had little experience dealing with all the new components. In desperation, we decided to start over with a new server build. The testing of the new server went OK, until I decided to throw another new wrench in at the last second – switching from MySQL to AuroraDB. “AuroraDB is 5X faster and 100% wire compatible with MySQL” according to Amazon, but it turns out that the PerconaDB client library on the server was not, the AuroraDB default parameter group is not configured properly for high query rates, and WordPress+mysqli PHP library+AuroraDB don’t play well together. So now, we had all our existing problems, plus the issue of configuring a new server a new set of management tools, plus the issues of switching to a new database server. Eventually, we solved all the problems we created by either learning to use the new components or reverting to old ones, but we never did figure out the cause of the performance issue, and simply patched it over with more hardware. What’s the moral of this story? If a website is suddenly slow, unreliable generally misbehaving for performance-related reasons, DO NOT TRY TO MAKE PERFORMANCE IMPROVEMENTS WITHOUT UNDERSTANDING THE ROOT CAUSE Any performance-related change should be tested to see if it makes things better. This is impossible without a stable site. Performance-related configuration and code changes should be based on evidence – quantitative proof that the specified change will help. Making changes based on hunches and Internet guides is a potentially endless process as software like MySQL, Varnish, Nginx, etc offer hundreds of parameters with millions of opinions online about what is best. The approach of making optimizations in the dark is a huge time drain when a quick and short-term solution is needed. You will make many changes with unknown effectiveness, possible falling into the dreaded Performance Panic Spiral of Doom: Try to fix a problem with guesswork without understanding the cause Break an unrelated component in the process Try a more drastic fix to fix both issues Repeat, until the site is a disaster zone HOW TO ACTUALLY RESOLVE PERFORMANCE OUTAGES: EVIDENCE-BASED ROOT CAUSE ANALYSIS Enhance your environmental awareness (by improving your monitoring & diagnostic tools*) until you can visualize, isolate, and identify the problem. Fix the problem. * For example, using New Relic, error logs, htop, ntop, xdebug, etc. performance tutorials How to regain access to an AWS EC2 instance that you’ve lost access to September 28, 2015 David Davidson Leave a comment Last night I accidentally locked myself out of a production EC2 instance. Arg! Panic! How I regained access: 1: Take a snapshot of the instance. (Note: if you require 100% uptime, this is a good time to restore the snapshot to a new instance and switch th...