Shambling Towards Perfection

The goal for sprint 7 is to create a git repo and finish the rewrite of the application, for reals this time. Including some of the items left over from last sprint, the sprint backlog items are: Priority Description Points 600 Set up git repository 2 590 Rewrite edit user/task/project code 4 580 Change email code as per discussion 3 570 Rewrite unfinished tasks page 3 560 Rewrite most recent tasks page 3 550 Rewrite blocking tasks page 3 500 Fix login error when user doesn't exist 2 This is a total of 20 hours, which, considering the time of year, seems more reasonable than 30ish. I'll also be doing some research on how to improve the task dependency/subtask relationships. Ideally I want to be able to display trees of these relationships (akin to comment trees on some websites), and this may require some database schema changes, all of which I want to get done in sprint 8 (before going live, basically). And that's it.

Yet again my biggest problem during this sprint was finding time to work on it. Considering the time of year, this isn't really a surprise. All the same, I had hoped to have more time than I did. During sprint 7 I'm going to try as hard as I can to work on the project at least a little every day. I've got a lot of stuff to do before next semester, so I want to get to doing it! I also thought of a number of things I had overlooked and discovered a few odd bugs, so the product backlog grew a bit during this sprint. I finished rewriting the new file link page (3 hours), the new user page (1 hour), the new task page (2 hours), new comment page (1 hour), the task page (3 hours), and I fixed the bug with navigating to the projects page (15 minutes). Some things took a good deal longer than estimated... There were no big issues during my development time, however I realized I had completely neglected to include time for making editing of tasks and projects work. Basically I wa...

Today I met with my adviser for the sprint 6 review and demo. It was pretty uneventful. I didn't get nearly as much done as I had hoped due to the busy holiday season (even programmers have friends and family, shockingly!), but what I did get done went well and turned out fine. All but the complicated  (recent activity/unfinished tasks/blocking tasks) read-only pages are done, and all of the creation code now works, so I demoed that. Then we discussed the project's email update capabilities, as I realized during the sprint that we'd never really talked about how that should work. If a user sets their preference to receive update emails, they get emailed when updates occur to their projects and tasks. I had originally set it up so that only the task creator and assignee would receive emails, but my adviser decided he would want emails as an instructor for every task update, even in the case of a student-led project where one student might create the task and assign it to a...

I've discovered I really hate testing. This is probably not a unique trait among developers. However it's really starting to be an impediment; I just don't feel like doing the testing which makes me not want to work on the project when that's the next thing I need to do. So I think I need to start procrastinating on testing, at least a little. I'm thinking I'll finish a couple of features, then test them all at once. Maybe have an official test day every other day or every third day or something. Of course this may backfire and end up with me frantically doing testing and the inevitable bug fixing all weekend long, but hey, at least the main development work will have been done. We shall see... (And yeah, obviously that whole test-driven development thing hasn't yet been implemented. I just don't know what I'm doing sufficiently to implement it, yet. Some day!)

The goal for sprint 6 is to finish rewriting the application, and to fix and prevent bugs and handle errors gracefully. There are quite a few items left from last time, so those are: Page Points Task 4 New User 1 New Task 1 New File Link 1 New Comment 1 Blocking Tasks 3 Most Recent Tasks 3 Unfinished Tasks 3 This adds up to 17 ideal man hours, which is a little low (hopefully; I like to try and stay optimistic!), so I want to add in a few more items to the sprint backlog. They are: Priority Description Points 510 Fix a bug with navigating to the projects page 0 (5 mins) 500 Fix the login error that occurs when the username doesn't exist in the user table 2 470 Wrap create statements and the fetching of the ID in one database transaction to prevent incorrect results 2 450 Update the application to handle errors nicely 5 This is another 9 hours, which brings us up to 28. I'm ever the optimist, so hopefully this will be the sprint where this is actually succes...

Yesterday I met with my professor for a demo of the pieces I had working; he was satisfied with it. Then we discussed my plans for the next sprint and through to the beginning of next semester. During the next sprint, I intend to get the entire application running again at the very least. I'm also hoping to start on the list of new features (discussed below) that I'd like to explore; my adviser was open to me doing any/all of them, although as he put it they're really for my own personal development/education rather than requirements from him. Of course the entire project is pretty much an exercise in personal development, since I already know how to make working software, but anyway. My goal with the project is to make an application that's as professional as possible, so I want to include some of the bells and whistles you'd normally expect from a commercial application. This includes things like resetting a forgotten password, locking an account after too many...

So on Sunday I finished sprint 5. It went fairly well, although I didn't have as much time as I like, despite it being a 3 week long sprint. I was out of town for 4 days, plus there was that whole Thanksgiving thing, so it came out to be more like 2 weeks after all. Considering the amount of time I had, I think it went ok. There were a few oddities and bumps in the road. First, I was highly annoyed when I discovered that in JDBC the table name/alias isn't included in ResultSet column names . Instead, you have to alias the actual column if you need to distinguish between column names in the result. Grr. Second, I discovered that it's really kind of a pain to call static methods using JSTL .  Since I use static methods in virtually every JSP to format the output for HTML/links/etc. to ensure user input doesn't do whacky things, this is a problem. For now I'm putting off rewriting the JSPs to use JSTL, because I'll have to write custom EL functions and maybe tw...

The goal for sprint 5 is the same as for sprint 4: to improve the database access code and the JSPs, as well as move Objects from the Session to the Request. The sprint backlog items, however, have changed substantially. For all of the following pages in the web-app, I need to rewrite the JSP and rewrite the servlet code; some of them have had their database code rewritten already but for the parts I didn't complete during the last sprint, I also need to rewrite the database code. These are really all at the same priority level, so I'm not bothering to give them a rating. Page Points Users 1 User 2 Task 4 Projects 2 Project 3 New User 1 New Task 1 New Project 1 New File Link 1 New Comment 1 Instructors 1 Application Configuration 1 Blocking Tasks 3 Most Recent Tasks 3 Unfinished Tasks 3 The total points for this is 28, so I think it should be do-able in 3 weeks.

I met with my professor on Monday and luckily he was very understanding about the lack of visible progress this sprint. I wasn't able to demo the entire system, only a few command-line tools to prove that what I did get done is actually working. But he was fine with my progress anyhow. We discussed when we would next meet, and decided that (since I'm crazy busy this week and will be out of town for a few days, as well) we'll meet the Monday after Thanksgiving. So this will be a 3 week sprint. I intend to push hard to finish this rewrite; since I will have a 5 day weekend for Thanksgiving I think it's do-able. This is the beginning of the end for the basic functionality of the system. I should be done by the beginning of the spring semester, leaving revising the UI and the paper as the main things to accomplish next semester. The paper shouldn't be much of a problem; I'm a pretty fast writer and way back in high school I was actually better at English than mo...

Sprint 4 is over. It was somewhat of a disaster. I completely and utterly underestimated the amount of time it would take me to rewrite the data access layer, application layer, and presentation layer. In retrospect it was insane to think it'd go that fast; this is a rewrite of almost the entire project, which I originally created over the course of 3 months. Whoops. Anyway I got through parts of the data access layer rewrite, and that's about it. I've increased performance in a lot of areas and improved the code overall. I'm now taking advantage of Java 7's try-with-resources statement, which is AWESOME and makes JDBC code much less hideous, and I'm no longer doing any catch (Exception e) but rather actually specifying the exceptions e.g. catch (SQLException e). I did also tweak my database tables, which necessitated rewriting my database creation and fake data creation code. I also wrote a test class to make sure the code I wrote is actually functioning pro...

The goal for sprint 4 is to improve the database access code and the JSPs, as well as move Objects from the Session to the Request. The sprint backlog items are: Priority Description Points 200 Move properties out of the session so that users can do things like have multiple browser windows open and bookmark pages 6 190 Improve database access code so that pages load faster 6 180 Refactor JSPs to use JSTL 8 170 Make all JavaScript and HTML fully standards compliant 8 I essentially split what I had been thinking of for the old "Move properties out of the session so that users can do things like have multiple browser windows open and bookmark pages" item into 2, with the other half being "Improve database access code so that pages load faster". This makes things more clear as to what exactly I intend to do. This is 28 ideal man hours; the next two weeks look less busy for me so I think I've got a good chance at making it this time. And so it goes.

Today I met with my advisor and we discussed the last sprint, including the issues I had. He decided I should try using the CS email system without authentication, and if that didn't work, the university's email system. I was able to do this during lunch and while the CS system still wouldn't work, I was able to send through the university's system, so that's what we'll be using. This makes the item Fix the email notification system so that it works with the CS department email settings finished, with around 2 hours of work. We also decided to backburner the attempts to get SSL redirection working from the root of the webapp, seeing as I can't figure out the problem and nobody on StackOverflow has even been able to help. As a last resort I can do the redirect programatically, but I would much prefer to do it via Tomcat's configuration if possible. My professor did seem generally pleased with my progress, and when I described the issues I had with PMD...

Sprint 3 is now complete. It went fairly well, although again I had trouble finding time to work on the project. There were some other frustrations which I'll detail below. I finished 3 backlog items: Run automated tools e.g. FindBugs to ensure code is correct and clean (8 hours), Write a shell script or similar to start the database server and tomcat (1 hour), and Fix the build.xml to only copy the correct jars to the web app's lib directory (15 minutes). Then there was the stuff I worked on a lot and didn't get done... I started on both Restrict the login page to only allow access via SSL and Fix the email notification system so that it works with the CS department email settings. For the former, I got SSL redirection working if the user navigates directly to the login page (e.g. http://localhost:8080/ProjectManagementSystem/login.html) but despite my best efforts I couldn't figure out how to get it to redirect if the user goes to the root of the web app (http://loc...

One of the issues for this sprint was improving my code's correctness. I installed and used the Eclipse plugins FindBugs , PMD , and CheckStyle . While these did point out some very valid issues, including not properly closing database resources, doing really bad things in an override of equals, etc., they also pointed out some things that completely flummoxed me. The biggest example is that apparently in Java one can declare virtually anything as final, including method parameters. I had known that methods and classes could be declared final though I had never used such functionality (nor had it explained or recommended either during my Bachelors degree or during my Masters degree, despite MANY Java-centric classes), but parameters was a new one. Some claim parameters should be declared final, while others say it clutters code for very little benefit. Additionally if you do, in fact, reassign a value to a parameter, the very same tools will mark it as an error. So there's qu...

Just for fun, and because I don't have a whole lot to report right now, here's my current home office set-up. These photos were taken after I'd gotten most of the product backlog completed, but before I started on any sprints. Whiteboard (made from shower board from Home Depot), bookcase containing all my computer science and math books, electric kettle for a steady supply of caffeine, and computer tower. Monitors, printer/scanner, plasma ball, headset, speakers, printer, and desk. Corkboard, small magnetic whiteboard, and plastic drawers for computer parts. Product backlog, sprint backlog, burndown chart, sprint progress board. Microwave, fridge, footlocker for non-perishable snacks and tea, and a second shower board whiteboard. Large calendar with which to plan sprints. So if the zombie apocalypse comes, I'll be able to survive for some time in my home office, and keep coding all the while. Nothing, dead or undead, shall keep me from fini...

Yesterday I met with my advisor to demo what I accomplished in sprint 2. Of course most of it was behind-the-scenes so there wasn't a lot to show, but he seemed pleased with my progress thus far. We discussed the security issues I've been researching and the insanity of trying to secure a web app in general, and he said security would be a good topic to discuss in the project summary paper thing I'll have to write. So I'm already getting a general idea of what will have to go in the paper, which is good. We also discussed what I hope to accomplish in sprint 3 and some plans for sprint 4. Right now there's a ton of steps involved in setting up the project for the first time (configuration files to edit and put various places, editing of tomcat's xml files, creating the database, etc. etc.) and so he'd like to simplify all that so that he can deploy my project on his own machine more easily. So we agreed that that will go in as an item for sprint 4. I'll c...

The goal for sprint 3 will be to make the project usable for my adviser and start making it more user friendly and start making the code more correct. There are several small tweaks I need to do so that everything will work correctly for my prof, so I need to do that, plus the FindBugs item from the last sprint. Code correctness and user-friendliness are very long-term goals, but this will be their beginning. The sprint backlog items are: Priority Description Points 300 Run automated tools e.g. FindBugs to ensure code is correct and clean 6 253 Write a shell script or similar to start the database server and tomcat 3 252 Fix the build.xml to only copy the correct jars to the web app's lib directory 2 251 Restrict the login page to only allow access via SSL 2 250 Fix the email notification system so that it works with the CS department email settings 4 200 Move properties out of the session so that users can do things like have multiple browser windows open and bookmark...

So I finished sprint 2 today. Yet again my biggest problem is finding enough time to put in as much work as I'd like. I suspect this will be an ongoing problem. I am mostly pleased with how much I accomplished though. I finished 3 backlog items: Fix fake data insertion code (1 hour), Add additional sanitization of input and other security measures (6 hours), and Add logging statements throughout the code (3). The only somewhat troubling aspect is that I'm still not 100% sure I've considered all the possible security issues. I've done a LOT of googling, but this information is extremely scattered. There's no comprehensive "Java web app security guide" that I've been able to find (though the OWASP stuff is very good and somewhat close). Mostly I've gone searching for one thing, and a random link or post will lead me to other things. It seems like you have to know what all the issues are in advance and then start searching for solutions to those iss...

As it turns out, security is actually harder than the internets originally led me to believe. Preventing SQL injection is as simple as making sure to use PreparedStatements (and not doing anything stupid like concatenating user supplied values with safe strings in the PreparedStatements), but preventing XSS is a whole 'nother kettle of fish. So I'm following the advice given on the linked site, and using their library to take care of escaping. I was using Apache Commons StringEscapeUtils for this but it only has one type of escaping for HTML, when apparently multiple different types are needed depending on where in the HTML the user-generated content is going. The Apache documentation doesn't really specify what it's doing, so I don't feel comfortable using it for either encoding inside regular HTML elements or inside attributes. I'm still using Java's own URLEncoder for encoding URL parameter values, as it seems to do the same thing as OWASP's UR...

I've been researching what's required to make my application less vulnerable to things like SQL injection or other malicious attacks. Imagine my surprise when I discovered that I've apparently already done the two biggest things, which are using PreparedStatements and escaping special characters . I've googled quite a bit and these are the main things I've found so far. According to the latter article I do need to do a few more small things, such as specifying the character encoding in my headers, but that seems to be about it... I'll be double-checking all my SQL to make sure there are no inappropriate uses of Statements, but apparently this issue is going to be a lot faster and easier than I expected. Rather shocking, really... Hopefully google hasn't lead me astray. In other news I'm leaning towards revising the issue "Add error checking, unit testing, and integration testing using JUnit, DBUnit, FindBugs, etc." to just focusing on Find...

I meant to mention this yesterday in one of my posts but forgot. I had originally downloaded jars for both slf4j and JCL , plus the jar that connects the two . I know I didn't do this for funsies; it was purely to get my code to compile so I could use the export-to-XML feature of DBUnit . However, I stumbled across a page that listed the dependencies of DBUnit and it didn't include JCL, just slf4j. I had updated all my jars so I figured maybe things had changed and removed the connector jar and the JCL jar... And poof, everything still compiled! Maybe I was somehow confused before (of followed bad advice on a blog/Stackoverflow post) and downloaded things unnecessarily, or maybe they've changed things in a more recent release, but either way I don't have to use both slf4j and JCL. Which is great, because when I tried to use them both the names of classes and the line numbers that were doing the logging got lost. Anyhow I now have things working very well with just s...

This was so unusual and exciting in a programmer-geek sort of way that I felt driven to blog about it. Today, for the first time in decade as a software engineer, I have actually used recursion in my work. It was in a Ruby on Rails model, to find all the descendants and parents of an object (there are no cycles in these relationships, so this works). And it was awesome. That's all. Just had to share that. ;)

The goal for sprint 2 will be improving stability and security (mostly a continuation of sprint 1, obviously). The sprint backlog items are: Priority Description Points 300 Add error checking, unit testing, and integration testing using JUnit, DBUnit, FindBugs, etc. 16 290 Add logging statements throughout the code 3 270 Add additional sanitization of input and other security measures 8 255 Fix fake data insertion code 2 The total ideal man hours is 29, which is hopefully a more reasonable amount for slightly less than 2 weeks (since I'm leaving a dayish for planning, setting up the demo and doing the retrospective). All of these carried over from the last sprint with the exception of the last item. I discovered during the last sprint that my fake data insertion code no longer worked due to changes I had made towards the end of the semester when I first developed the application. I really need those routines in order to have adequate data for testing, so they need fix...

So last night I started thinking about sprint 1 and how it went, what I'd do differently etc. for the sprint retrospective. The biggest thing I learned is that there's no way I can put in as much time each week as I'd like to on the project. I would love to put in 20 hours per week but that's just not going to be possible while maintaining my sanity. The biggest problem is that weeknights quickly get crowded, so I can really only get in a half-hour here or an hour there M-F. As such I need to concentrate on getting a lot done on the weekends. I'm going to start designating one day per weekend as a project day and do no other activities on that day. This should help a lot, I hope. The main thing I will be sure to do differently is to set up the demo much earlier. I waited until Sunday night and then ran out of time. Next time I'll set it up in the morning/early afternoon of my designated project day in case something goes wrong. I'll officially end my sprin...

Yesterday I had my meeting with my prof. Alas, I had some configuration problems on the department machines (Tomcat wouldn't pick up my JRE_HOME setting for unknown reasons) so I wasn't able to demo during the meeting, but I was able to get a demo up and running during my lunch break. The demo included SSL for login, hashed passwords in the database, restricting access to the HSQLDB to the same server it's running on (as far as I can tell; I haven't figured out how to test that one yet), and logging. My prof seemed satisfied with how much I've done so far and we discussed my plans for the next sprint (mostly finishing what I didn't finish this sprint), so the Sprint Review, such as it was, went well. During the meeting I also discovered I won't have to do a written exam on the classes I've taken, contrary to the impression I had gotten from the department webpage, so a big hooray for that. A "project report" describing the project, how I did ...

So apparently slf4j is a logging abstraction framework meant to let you use the actual logging implementation of your choice. So slf4j lets you use Jakarta Commons Logging (and this is what DBUnit does), which is ALSO a logging abstraction framework meant to let you use the actual logging implementation of your choice. So I STILL need yet another library in order to get logging to files instead of to System.err. I just... For crying out loud. Fine then. I just downloaded log4j . Are you finally satisfied, logging libraries??? Argh. :(

I met with my adviser for the first time last Wednesday, and we decided our next meeting would be September 24 and then we'll meet every two weeks thereafter. Since the 24th would fall in the middle of a sprint I decided to extend the first one to end this Saturday, the 29th (I was also under the weather several days last week, so it was a good move all around). On the 30th I'll upload all my code to the version control system my prof has set up (svn) and configure everything so that it will run on the department servers. I'll be demoing my latest work at every meeting, so every other Sunday will be configuration day. I realize continuous builds on the department machines would be better but it's not exactly practical, so I'm settling for continuous builds on my home machine. Since my last post I have successfully configured Tomcat to use SSL and jbcrypt . The latter involved writing a custom realm but turned out to be quite trivial; I simply extended JDBCRealm ...

From the lack of updates you can probably guess that I didn't accomplish much this summer. Life got in the way as I had too much else to do. But now the fall semester has started, I'm officially registered for just 3 credits for my Masters project class, meaning I'll be doing another 3 in the spring and graduating in May instead of December, and I'm back at work on the project. I finally started sprint 1 on September 1. So far I've primarily been researching for the "restrict access to the HSQLDB" and "change the way passwords are stored and transmitted" backlog items. After vast amounts of googling I've discovered how to do these things and for the second decided on which methods are needed. I'll be using SSL for just when the user logs in and the bCrypt algorithm as implemented in jBCrypt to hash passwords before storing them. Other than passwords there really isn't any sensitive info in the app; if people didn't reuse pass...

The first sprint, which I will hopefully start tomorrow, has a goal of improving the application's overall security and reliability. Right now I have some pretty egregious security violations, such as plain-text passwords, and most errors get swallowed, so it's very difficult to tell what the problem is when something has gone wrong. So I need to start remedying that. The sprint backlog items I have selected are: Priority Description Points 300 Add error checking, unit testing, and integration testing using JUnit, DBUnit, FindBugs, etc. 16 290 Add logging and propagate exceptions instead of swallowing them. 8 280 Restrict access to the HSQLDB to the server on which it is running. 2 270 Add additional sanitization of input and other security measures 8 260 Change the way passwords are stored and transmitted so that they are encrypted 4 The sprint will be 2 weeks long. My story points are sorta kinda "ideal man hours" but really are more relative to each ot...

I have finally overcome some technical difficulties, including finding a CS department machine that was actually up so I could retrieve info I needed that had been stored with my original code, and remembering that I had installed 32-bit Eclipse on my machine originally, because 2 years ago when I set it up Eclipse didn't play well with 64-bit Java. The latter resulted in my shiny new 64-bit Java 7 not working with the old version of Eclipse. Getting 64-bit Eclipse in the latest version (Juno) solved that problem. All the technologies, libraries, plug-ins, etc. I need for the project are now fully updated to the latest versions, and I was able to get my project running on my home machine (using Tomcat as the application server). It had been so long since I'd set things up that I'd forgotten how, so I did prove that the instructions I wrote up for my professor to follow worked very well. Go me. I have also been looking more into Java libraries that may help with variou...

I've been working on my Masters degree one class at a time for the last 4 years (working full-time doesn't leave much time for either classes or homework). In May I finally finished up all of the required coursework, so now I just have my project left; that will be the (initial?) focus of this blog, as I created it solely because of the project.  I hope that by documenting my process, design decisions, etc. here, it will serve as a head-start on the required paperwork and documentation for the finished project. The project is called, for now, the Project Management System, and is based off of a project I completed for Advanced Software Engineering in the spring of 2011. It's a web-app using HSQLDB as the database, Java for the data access layer, Java Servlets for the application layer, and JSP/CSS/JavaScript for the presentation layer (to the latter, I intend to add JQuery and JSTL in the near future). A presentation  on my project as it stood at the end of the class is a...