Comments (6)

By Mark Curphey, January 24, 2012

If you are a regular reader of this blog you should know I am working on a book for O’Reilly called Practical Software Security. It’s in the early stages and evolving as we start to get into the details. You can sign up to get notifications of progress here.

There are five main sections:

• Introduction
• Security Concepts
• Languages & Frameworks (was called Tools & Technologies)
• Building a Software Security Program
• Engineering Scenarios

In the Security Concepts section we will introduce developers to things like cryptography, authentication, authorization and then in the Languages & Frameworks sections we will cover the security features available and how to use them properly for the popular development technologies. In the engineering scenarios well get to code level guidance of how to pull it all together to solve real world problems that all developers face like securing a REST API or creating a federated authentication system.

I am delighted that we have been able to recruit a fantastic list of subject matter experts to write and review the Languages & Frameworks section of the book. I will be writing the Security Concepts section (and the Ruby on Rails section) and the following folks will be writing or reviewing:

• HTML5 (and friends) – Mark Curphey
• Java – Pravir Chandra
• .NET – David Rook (Security Ninja + Microsoft MVP for Developer Security)
• JavaScript (Client + Node) – Gareth Heyes (The Spanner) with Rey Bango as a core reviewer (JQuery Core Team Member)
• Ruby on Rails – Mark Curphey with Heiko Webers (wrote the official Rails security guide) as a core reviewer
• PHP – Mike DeLibero with Chris Shiflett (wrote the O’Reilly PHP Security book) as a core reviewer
• iOS & Android – Dan Cornell
• Identity – Gunnar Peterson

Note: we will be doing C / C++ as there is still so much being produced but haven’t yet locked on that author and we will be figuring out how to deal with things like Spring or Cake (and where to draw the line).

In the Building a Software Security Program section we organize the section into People, Process & Tools. Justin Collins (author of Brakeman Scanner) is going to write the static code analysis section in Tools and Tasos Laskos (author of Arachni) will write about how dynamic web application scanners work. Many other tools will of course be covered! I will be writing extensively about integrating Agile practices and using TDD / BDD techniques and tools. I probably won’t start on this section until March / April. I am hoping we will have the Security Concepts section and the Languages & Frameworks section complete by the end of February so we can open up a site for a much broader set of reviewers (invite only but register to get on the invite request list here) around March.

OK, now to set up the git repo for the contributing authors, create a README.md for them and kick off the discussion about we want to structure things. Gentlemen start your engines!

Edits : 1/25 – Added HTML5 (and friends), 1/25 – Added Gunnar Peterson to Identity

Comments (3)

By Mark Curphey, January 17, 2012

Create Local Repository from an Existing Local Project

Comments (3)

By Mark Curphey, August 24, 2011

This is my tickler list of web dev stuff I am currently watching and or playing with :

• CoffeeScript – http://jashkenas.github.com/coffee-script/
• Lettuce – http://lettuce.it/
• Selenium – http://seleniumhq.org/
• Jasmine – http://pivotal.github.com/jasmine/
• Saas – http://sass-lang.com/
• Compass – http://compass-style.org/
• Jenkins – http://jenkins-ci.org/

What’s on your list (and missing from mine)?

Comments (2)

By Mark Curphey, August 17, 2011

This is part 4 of a series on setting up a great Python development environment on OSX. Part 1 can be found here, Part 2 can be found here and Part 3 here. This simple part focuses on configuring Pycharm to use your VirtualEnv (Part 3).

The religious debate over IDE versus text editors can rage on. I like both but generally want to use an IDE where it helps me. The JetBrains IDE’s are great and Pycharm is the Python version. It’s $99 but well worth the money. Current version is 1.5.x and can be found here. Installation is simple and easy.

The first thing you will want to do when you are installed is to be able to point your project to the virtual environment you created in Part 3. Pycharm stores IDE preferences in a series of files in the.idea folder. These is a hidden folder so you need to use a neat OSX trick to view them. You can set global OSX preferences to show all hidden folders and files or as a quick temporary fix you can hold down Shift + Command + Period at an Open dialog box! Create a new project (in my case “test_charm”, navigate to it from File Open in Pycharm and then Shift + Command + Period and you will see the files in my screen shot.

Some settings are IDE wide and some settings are per project. I haven’t audited which ones are which (if anyone has done this it would be useful so please share) but I am assured by the JetBrains support that the interpreter configuration is per project (so I assume stored in the.iml file) and I have tested this to validate. By default you will see below when I create a new project I am using the system interpreter (2.7.1) and you will notice I just have a few site-packages installed.

To set up Pycharm to use use a VirtualEnv I first create the virtualEnv in terminal (see Part 3), then create a new Pycharm project in the folder. The virtualEnv creates three folders (bin, lib and include so I create a “source” folder at the same level and create the project in that.

Now create your new project in the “source” folder of the virtualenv structure

From the Pycharm preferences menu select the Python interpreter and navigate to the /bin folder of the virtualenv you created. Select add and Pycharm will do some analysis. When it has finished remove the system interpreter and you will be left with your virtualenv.

Back in the main Pycharm window you all now see the project is pointing at the virtualenv interpreter! Voila! You can install packages using pip etc. and away you go !!

Next in the final part of the series an end-to-end project setup using all thats we have been threw for a simple web app using Django and a SQL Lite database along with Splinter, Selenium, Lettuce and South!

Comments (28)

By Mark Curphey, July 16, 2011

@curphey on Twitter and on Google Plus

I took a week off of work this week to learn Python. Truth is I have been trying to learn Ruby (and) or Python on and off for the last 18 months or so but work and kids (and the “dog ate my homework” syndrome) have always found priority and I made little meaningful progress. After a big release at work I checked my vacation balance, bit the bullet and booked this week off work. I told everyone I was going dark. It’s Friday and so far I am feeling like I have really got to grips with the basics of the core of the language. I am now in a place where I can be productive writing code and solving my own problems. I think I now have the basic level of knowledge to continue learning and overtime actually become a half-decent decent Python developer. In this post I share how I approached getting my results so far.

If you want to know why I wanted to learn a language and why I want to learn Python in particular there is of course a short story. I got involved in computer security via formal education (cryptography and mathematics). As my career progressed through start-ups and financial services companies I became good at managing technical projects and managing technical teams. Through personal interest I steered towards running software security programs and running teams of developers building software security tools. For the last five years I have been more interested in “software” than “software security” and while I am no pointy-haired boss (I have picked up a lot along the way) I have never been formally trained as a computer scientist or worked as a commercial software developer (code contributing developer). When I turned forty, two years ago I had the classic mid-life crisis. I took up distance running and evaluated my long term career plans. Many people start off their careers as highly technical individual contributors and slowly morph into spreadsheet crunching people managers. Power, money and (pseudo) respect in most corporate structures is understandably aspirational for some people but having been there and done it, I made a conscious decision to morph my career back the other way. I won’t ever stop wanting to lead teams of smart motivated and passionate people but I want to be a grass roots technical contributor creating meaningful software. I could write a very long post indeed on how I think the very nature of software teams is changing way beyond Agile and how I think the future is very small self-contained teams operating in an eco-system but perhaps for another day. The short summary is that I decided I wanted to be the member of a rock band and not a member of an orchestra and want to lead software teams from within.

Why Python? With the huge choice of languages and having explored a number over the years (Java, Ruby, Perl, C#) I decided to pick one with a view to becoming proficient in it rather than try and become a “jack of all trades” in a few. I view JavaScript and CSS as base knowledge for any modern developer so didn’t include those in the scope of my decision. You have to know them regardless if you want to build web applications. I have always been passionate about open source software and so one pre-requisite that there was a strong open source community (online, local meetings etc.) and eco-system (tools, components etc.) behind my language choice. At college in ’97 I first played with Java back when it was all about applets, then later when I ran the software security program at Charles Schwab between 2001 and 2003 we were all about EJB so Java might have been a natural choice and the eco-system is clearly very strong, but for some reason the community around Java doesn’t excite me. For me it essentially boiled down to Ruby vs. Python and to cut a long story short Python won. While Ruby has stronger test support with Cucumber (and friends) and of course a first class web framework in Rails, the developer community felt arrogant. Hang out on a ruby language mailing list for a week and you will know what I mean! Python on the other hand has a very strong scientific community behind it with a wider range of libraries for a broad spectrum of science things like bio-informatics. There is a great MVC framework called Django and in recent years Google has adopted it as one of its formal language for both use internally and on the Google App Engine. There is no real science behind my choice and it is very much personal preference but simply put it feels like a decent choice at this time.

Of course there will be a lot of code, a lot of learning and a lot of time before I will be a competent developer, if you are like me and want to learn Python (and have a day job that seems to get in the way) here is what has worked for me that you may want to consider.

1. Book a dedicated week – This is by far and away my number one tip. I took last Friday off to deal with the inevitable bow-wave of mail from work so I could enter this week 100% clear of any work issues. I structured my day with a 5 – 6 mile run every morning (obviously optional) followed by 4 hours of dedicated coding time. I take a break for the afternoon and then follow-up in the evening for a few hours (no fixed length) to cover any loose ends, extra reading and preparing for tomorrow. I created a bow-wave of learning so I wasn’t going into this week cold by starting tutorials a few weeks ago. This allowed me to focus this week on exploring important concepts and not understanding basic theory. I knew what I needed to learn going into this week and what success looked like for me so created my plan for the week (see below) ahead of time.

2. Agile Planning – I scoped out my work using Agile planning techniques. This maybe overkill but got me into the right mind-set. I used the community edition of Rally and set up a backlog and daily iterations. I even entered exercises as tests to prove my knowledge to myself. I have been able to look at my backlog, add stories and tasks and get better at estimating my velocity as the week wen’t along. Other decent free options for this include Pivotal Tracker or Agile Zen.

3. There is no substitute for writing code – You can read as many book as you want and write code in your mind but there is no substitute for writing code on your computer. It really is the only way you will learn syntax, semantics and how things really work. Don’t fool yourself into reading a chapter of a book or watching a video, writing code is what teaches you to write code. There is nothing more to say on that topic. Zed Shaw’s “Learn Python the Hard Way” echoes this. Use the Python shell as much as you can to experiment. iPython gives you a great extended shell to play with.

4. Hire (pay) a good instructor – Having a good mentor / instructor is essential. I was lucky enough to have a very smart friend (who is very, very good developer) last year agree to meet me for coffee every other week and teach me some Ruby. I never really made progress. Reason? I wasn’t committed. I never had to pay him, I would meet and catch up on a social level and our relationship was based on friendship and not on learning. This time I did some Googling, found out that the local University (University of Washington) ran an extension class and emailed the lecturer asking for private lessons. I met him for coffee first to make sure there would be a social fit and that his style would match what I was looking for. It’s not cheap (although all up will probably be 50% of the cost of doing a formal class like this) and has allowed me to operate at my own pace and on topics I want to cover. We can get side-tracked when we want. The lecturer (Brian Dorsey) is great and I look forward to meeting. During this seven day sprint we planned to meet three times and every Friday for a few weeks before and afterwards.

5. Use the Right Learning Material – Over the years I have bought a lot of language books. They range from step-by-step instructional books like the APress or Head First series to less structured books like the O’Reilly language series. While I have gotten a lot from them I have either found them to be “draw by numbers” or not organized in the way I wanted to learn. Bryan suggested “Python for Software Design – How to Think Like a Computer Scientist” which is what the UW course is based on. You can get a free PDF version here. The book has been a perfect fit for me as it talks about computer science in terms of recursion and stack diagrams and has exercises to build fractals and solve math problems. It is an an academic level that I can relate. The right material has really had a significant impact on my learning this week.

6. Use an IDE – Many purists will expose that using a text editor like TextMate (which I own) or TextWrangler is the best way to learn but for me using a fully fledged IDE has been a serious benefit. I use PyCharm and for the most part love it. When you are exploring an API you can see the functions it supports and syntax errors are highlighted as you type. While this could indeed make you lazy I think it helps you get up to speed faster so that the syntax becomes second nature. For me it’s like the difference between using a text editor or word processor to write a story.

7. Use good development practices as you learn – All week I have put my code into revision control. I generally use Git so also took the opportunity to learn Subversion this week (my only real violation of my next tip but it was so simple that …). I comment all my code no matter how trivial the example and even took to running it through the Python style guide PEP-8. Towards the end of the week I have even taken to writing unit tests for all code. Python Koans are great for learning testing in Python.

8. Just Learn the Language, Nothing Else – It’s very tempting to also learn Django at the same time as learning Python. If you want to build web apps you will need a framework to build on but I think it has been very valuable to separate the two. Apart from separating the complexity and scope it forces you to focus on the language only and not how to use the language to drive a framework.

That’s it for now. If I can think of more I will add it and I would love to hear others tips and tricks.

Useful References

A Jump Start for Learning Python

33 projects that make developing django apps awesome

http://learnpythonthehardway.org/

ipython

Python Koans

Comments (0)

By Mark Curphey, December 28, 2010

I have been looking for a meaningful new side-project for a while and have now been actively working on an idea with a “partner-in-crime” for a few months. We plan to launch a web-site sometime early in the new year and we will start blogging and tweeting about it soon.

The last big side-project I started was OWASP (which now has thousands of “partners-in-crime”). Looking back the time I spent working on OWASP was one of the most motivated and creative times of my adult life. When we first started the project there was no goal or rule book to follow. There was a set of ideas and a collective passion for web application security. That was it. Everything else got figured out along the way as a community. A community that had no definition, no initial structure and no governance. I don’t think anyone even recognized it as a community for a year! Probably most importantly was that there was a feeling that there were no-rules and no-limit to what could be achieved if a few like minded passionate people came together. While I haven’t been involved in OWASP in any meaningful way for a number of years and so can’t take credit for it’s phenomenal success over the last decade, it ignited my interest in something that I think is quite profound. In truth while there will always be software security in my DNA it was the community aspect of OWASP that I enjoyed and learned from the most. I read a lot and often find myself relating passages in modern online social science books like Getting Real (Re-Work), The Long Tail, Groundswell and Here Comes Everybody to what I have observed happened (and is happening) at OWASP. It has been a very good online social science lab!

Of course at the time certain patterns of behavior or actions weren’t calculated. They just happened by luck (lucky timing  I guess) but several years on and way too many hours spent online I am utterly convinced that there is a very strong correlation between key patterns in online social science and organizational theory which when combined with design patterns in social software are strong indicators of the success or failure of online communities.

The way people organize themselves, the types of people that are involved, who makes decisions (and of course how decisions are made), how community members are recognized and rewarded and how disputes are resolved are all critical patterns. The type of software communities embrace is also vitally important to success. Social software be it a wiki enabling collective editing, a mailing list enabling seamless discussions or a blog enabling friction-free publishing work with different degrees of effectiveness depending on the community.

Most fascinating of all of this of course is that no one size fits all. There are clear patterns but not play-books and that’s what makes it so fascinating.

So my next project will a community for community organizers and developers!

- Mark Curphey

Comments (1)

By Mark Curphey, September 7, 2010

I have an interesting personal project brewing looking at ways to build better community software and am exploring emerging thinking about the next generation of web applications. Tim O’Reilly wrote a seminal blog (part one and part two) describing the Internet as THE new operating system. In the post he uses the analogy of traditional operating systems like Windows or Unix and the computing services they provide (file storage, networking, user management etc.) and then describes the Internet as an Operating System providing information services to distributed applications. There are some obvious fundamental differences in thinking about building scalable applications on the Internet Operating System such as binding together distributed services. There are also a number of less obvious but fundamentally important things to consider such as the fact that there will be many choices for common services such as authentication (Google, Yahoo, LinkedIn, Twitter, FaceBook Connect etc.) and smart application designers will need to adopt new architectural and functional models that provide the user with their preferred choice and not just one option as is the norm with traditional operating systems.

As I have started to think about the model I started to capture the potential services you could consume in a diagram (below). Its a first pass that I plan to update. The simple rule I am operating under is that to be considered a service there should be a published API in which to connect.

What else would you include?

Comments (0)

By Mark Curphey, September 7, 2010

This post is #10 (the last and hopefully the best) in a series 10 Crazy Ideas That Might Just Change the State of the Security Industry.

I have been doing Agile development in earnest for about two years now. First while building security tools like static code analysis and web protection libraries and most recently building large scale web sites. I have learnt a lot (actually I have learnt a great deal) and while Agile is clearly not a panacea, I firmly believe that it could be successfully applied to many security situations to improve the state of the security industry. What follows are a selection of ideas; certainly not complete and certainly not exhaustive. This post should not be confused with infusing security into an Agile process. This post is about using Agile techniques for security.

For context Agile works best in a world sandwiched firmly in-between complex & chaotic projects and predictable & simple projects. This is the Agile “sweet-spot” and luckily enough (from my experience) where 99% of the security projects I have seen fall. Many stalwarts who don’t understand Agile often say something like “But Agile is not for the Enterprise.” Here is a factual answer: “The Enterprise is a colossal spaceship from the 80’s flown by people with bad taste: Highly advanced, but purely fictional”. Don’t make the common mistake of confusing the size of a project with complexity or predictability!

Agile takes a pragmatic approach to life by acknowledging that when we first encounter a project (or topic) it is the point when we understand the very least about it. This is called the “Cone of Uncertainty”. What tends to happen in non-Agile environments is that people try to estimate the size of a problem or project at the funnel end the very point when they are least qualified to make an accurate estimation. This is true for example of security reviews. A security consultant typically is given an overview, an RFP or high-level description and asked to estimate the amount of time and therefore cost a project will take. They are asked to estimate the time and cost to perform a function such as a code review based on nothing more than a high-level description of the application or in some cases basic information like the LOC. In reality what happens (and this maybe a dirty little truth about the industry but I can tell you that its the norm) is that consultants will simply cut back on the scope or depth of analysis to match their initial estimation. Instead of getting an analysis of the things that need analyzing you end up getting an analysis of the things that can be analyzed in the amount of time that was guessed. This is actually why the concept of Threat Modeling is powerful. It allows an analyst to understand the system and isolate the areas of importance that warrant further detailed investigation.

Another common Agile concept is ‘Plan for Today And Worry About the Future Tomorrow’ – Most people try to plan for the future and layout elaborate timelines that stretch into the distance. Most are based on what-if scenarios. What-if “X” happens or what-if “Y” happens ? The reality is most people fail with this approach. Most plans are nothing more than guessing games. Security assessments and architecture reviews usually do exactly this, guess what will need to be done. Agile promotes worrying about real tangible problems that we face today and not getting side-tracked into thinking about issues that may or may not occur tomorrow. Don’t be fooled into thinking that this means ignoring architecture and scale. It just means keeping a solid eye on reality and not on fiction.

And for people who think that accurate estimation is complex and really tough try this. Choose a city about an hour away, then choose a city a few hours away and then choose a city in the middle of the country and then one of the other side. Get a team of people to estimate the time it will take to travel to each location using relative measurements ie if the city one hour away is one then how many units would the next one be? We did this exercise in Seattle with Tacoma, Portland, San Francisco, LA, Las Vegas and Miami and were able to estimate the distance from Seattle to Miami within 100 miles! That kind of result is repeatable and predictable. Most people spend way too much time trying to make perfect estimations when in reality accurate approximations using comparative techniques are all that’s needed.

The Agile manifesto http://www.agilemanifesto.org lays out a set of principles in a particularly meaningful way and one such principle is that Agile values ‘Individuals and Interactions over Process and Tools’.  Another way to say that is that partnerships drive results. In my experience very true partnerships between security specialists and developers exist. Instead we have lots of CYA process and “gotcha” type traps built and operated. Security people must partner with developers and encourage them to partner back.

The Agile manifesto talks about ‘Working Software over Comprehensive Documentation” & about ‘Customer collaboration over Contract Negotiation”’ – I will always remember when I was audited by a PCI firm that shall remain nameless (to protect the stupid). I had some code that protected a token passed on URL parameter. It was cryptographically secure (from replay, tampering, viewing etc) but I failed the audit as the token wasn’t passed over SSL. I was caught on a technicality. This was a case where a secure system (working software) didn’t meet a checklist (comprehensive documentation). As Forest Gump would say “Stupid is as stupid does”.

Finally (for this post anyways) Agile promotes the mind set of always working on the highest priority thing in small increments to deliver the greatest value while embracing change. I believe that by adopting the way we manage backlogs would help security teams groom what’s important and always be working on what’s the most important thing to be working on at that time.  In Agile we maintain what is called a Product Backlog, a fancy name for a list of things we want to do. When we understand enough about each item on the backlog (usually to estimate the effort to take on an item) we rank them and work from the top of the pile. We spend time and effort refining the backlog, an activity called Backlog grooming. Compare that to the way a typical security assessment is done today, procedurally following a long list of pre-determined things. By spending more time on the important topics as we define and understand them would allow us to be more effective at reducing real risk.

I hope the series has been useful.

Comments (0)

By Mark Curphey, September 6, 2010

This post is #8 in a series 10 Crazy Ideas That Might Just Change the State of the Security Industry.

When Jason Fried of 37 Signals spoke at Web 2.0, most for the media picked up on a great meme proposing that web site owners should think of themselves as software curators. Instead of trying to pack in as many features as possible, it’s is the ‘collection’ of features that create an experience and what ultimately matters. During the talk he spoke about design driven software and used a few great analogies, one using a clear water bottle to explain why it’s easy to assess the quality of the design of a physical objects and then went on to discuss why software is so relatively hard. I recommend the video here and a talk by Ryan Singer the 37 Signals UX person here.

End users of most software are simply confused about security warnings and therefore generally ignore them. Security features such as authentication systems are privacy settings are complex and in general seem to be designed by and for developers. FaceBook had 4,000 comments in a week for privacy features that it claimed already existed.

Most security software is also nothing more than a hodge-podge of every possible feature every possible security person can think other users might want want. Security people have long ridiculed the desire to have a “shiny red security button” including myself many many times. The truth is that security is way too complicated for the majority of people and it has to be made easy. That wil only come by embracing Design Driven Security.