Defeating modern attacks – even distributed ones – isn’t the problem. The problem is detecting them in the first place.

Last week researchers claimed they’ve discovered a way to exploit a basic security flaw that’s used in software that’s in high use by Web 2.0 applications to essentially support if not single-sign on then the next best thing – a single source of online identity.

The prevalence of OAuth and OpenID across the Web 2.0 application realm could potentially be impacted (and not in a good way) if the flaw were to be exploited. Apparently a similar flaw was used in the past to successfully exploit Microsoft’s Xbox 360. So the technique is possible and has been proven “in the wild.”

 The attacks are thought to be so difficult because they require very precise measurements. They crack passwords by measuring the time it takes for a computer to respond to a login request. On some login systems, the computer will check password characters one at a time, and kick back a "login failed" message as soon as it spots a bad character in the password. This means a computer returns a completely bad login attempt a tiny bit faster than a login where the first character in the password is correct.

[…]

But Internet developers have long assumed that there are too many other factors -- called network jitter -- that slow down or speed up response times and make it almost impossible to get the kind of precise results, where nanoseconds make a difference, required for a successful timing attack.

Those assumptions are wrong, according to Lawson, founder of the security consultancy Root Labs. He and Nelson tested attacks over the Internet, local-area networks and in cloud computing environments and found they were able to crack passwords in all the environments by using algorithms to weed out the network jitter.

Researchers: Password crack could affect millions

ComputerWorld, July 2010

Actually, after reading the first few paragraphs I’m surprised that this flaw wasn’t exploited a lot sooner than it was. The ability to measure fairly accurately the components that make up web application performance is not something that’s unknown, after all.  So the claim that an algorithm can correctly “weed out” network latency is not at all surprising.

But what if the performance was randomized by, say, an intermediary interjecting additional delays into the response? You can’t accurately account for something that’s randomly added (or not added, as the case may be) and as long as you seeded the random generation with something that cannot be derived from the context of the session there are few algorithms that could figure out what the random generation seed might be. That’s important because random number generation often isn’t and it can often be predicted based on knowing what was used to seed the generator. So we could defeat such an attack by simply injecting random amounts of delay into the response.

Or, because the attack depends on an observable difference in timing, simply normalizing response times for the login process would also defeat this attack. This is the solution pointed out in another article on the discovery, “OAuth and OpenID Vulnerable to Timing Attack”, in which it is reported developers of impacted libraries indicate that just six lines of code will solve this problem by normalizing response times. This, of course, illustrates a separate problem, which is the reliance on external sources to address security risks that millions may be vulnerable to now because while it’s a simple resolution, it may takes days, weeks, or more before it is available.

This particular attack would indeed be disastrous were it to be exploited given the reliance on these libraries by so many popular web sites. And though the solutions are fairly easy to implement, that isn’t the real problem. The real problem is how difficult such attacks are becoming to detect, especially in the face of the risk incurred by remaining vulnerable while solutions are developed and distributed.

DEFEATING is EASY. DETECTING is HARD.

The trick here, and what makes many modern attacks so dangerous is that it’s really really hard to detect them in the first place.

Any attack that could be distributed across multiple clients – clients smart enough to synchronize and communicate with one another – becomes difficult to detect, especially in a load balanced (elastic) environment in which those requests are likely spread across multiple application instances. The variability in where attacks are coming from makes it very difficult to see an attack occurring in real-time because no single stream exhibits the behavior most security-focused infrastructure watches for.

What’s needed to detect such an attack is to be more concerned with what is being targeted rather than by whom or from where. While you want to keep track of that data the trigger for such brute-force attacks is the target, not the client activity. Attackers are getting smart, they know that repeated attempts at X or Y will be detected and that more than likely they will find their client blacklisted for a period of time (if not permanently) so they’ve come up with alternative methods that “hide” and try to appear like normal requests and responses instead. In fact it could be postulated that it is not repeated attempts to login from a single location that are the problem today, but rather the attempt to repeatedly login from multiple locations across time that’s the problem. So what you have to look for is not necessarily (or only) repeated requests but also at repeated attempts to access specific resources, like a login page.

But a login page is going to see a lot of use so it’s not just the login page you need to be concerned with, but the credentials, as well. In any brute force account level attack there are going to be multiple requests to try to access a resource using the same credentials. That’s the trigger. It requires more context than your traditional connection or request based security triggers because you’re not just looking at IP address, or resource, you’re looking deeper and trying to infer from a combination of data and behavior what’s going on.

THIS SITUATION will BECOME the STATUS QUO

As we move forward into a new networking frontier, as applications and users are decoupled from IP addresses and distribution of both clients and applications across cloud computing environments becomes more common, the detection of attacks is going to get even more difficult without collaboration.

The new network, the integrated and collaborative network, the dynamic network, is going to become a requirement.  Network and application network and security infrastructure needs a new way of combating modern threats and their attack patterns. That new way is going to require context and the sharing of that context across all strategic points of control at which such attacks might be mitigated.

This becomes important because, as pointed out earlier, many web applications rely upon third-party solutions for functionality. Open source or not, it still takes time for developers to implement a solution and subsequently for organizations to incorporate and redeploy the “patched” application. That leaves users of such applications vulnerable to exploitation or identity theft in the interim. Security infrastructure must be able to detect such attacks in order to protect users and corporate resources (infrastructure and applications and data) from exploitation while they wait for such solutions. When is more important than where when it comes to addressing newly discovered vulnerabilities, but when is highly dependent upon the ability of infrastructure to detect an attack in the first place.

We’re going to need infrastructure that is context-aware.

I’ll be speaking on a panel at Cisco Live on July 1.  I’m looking forward to talking about the new demands on network infrastructure, and whether or not the enterprise is ready for seamless cloud.  Frankly, so much of the discussion about cloud is for SMEs (or regarding apps) and so little is about the readiness of cloud for the enterprise that it is refreshing for Cisco Live to embrace this topic.

Even the mention of “private cloud” gets negative reactions from some of the clouderati.   I heard: “No such thing” yesterday on a cloud pundit call.  Yet at the end of the day enterprises will be assessing when, where and what can be delivered from any cloud versus a private cloud and the answers will have a significant impact on the evolution of cloud computing.

While I think Amazon and Google have done well delivering undifferentiated services via subsidized business models, it is fair to ask when and how can enterprises take to the clouds.  IMHO it’s when the network is ready.

You can view the session abstract here: Seamless Enterprise Extension to Cloud (SEEC) - Ready for Primetime?

Read more...

As I’ve mentioned, Cisco’s CTO predicted 1 trillion net connected devices by 2013 and I don’t think Padma was counting virtual machines.  With the rise of the three horsemen (netbooks, virtualization and cloud computing) it seems obvious that today’s enterprise networks are in need of a fundamental overhaul, starting with the automation of the tired, manually managed IPAM, DNS and DHCP infrastructure.

Read more...

I wouldn't even be a member of the Infrastructure 2.0 blogging group if it wasn't for the fact that I love the opportunity that cloud presents to enterprises. I know, where's the "but"? Keep reading....

But, as much as the benefit of cloud is relatively well understood, like any complex activity you can't just plug it in, turn it on and assume everything will work out just fine. I've talked to a number of people over the last few years who've complained that projects to implement virtualization were complex or difficult and I couldn't agree more. Here's the rub, every change we ever make, in life, IT or otherwise is difficult and often complex if it's worth pursuing. There are few people if any who can say that the well executed implementation of virtualization hasn't been a boon to enterprises all over the world, even though they were complex & often difficult.

So how does foreseeing this problem created by dumping cloud into our environments early help you develop a strong team and an excellent service?

Introducing cloud to your environment without a strategy of ownership could set back your efforts for years. At many levels cloud will change the very nature of how we deliver services, even how we own or don't own hardware and data centers. My belief is that at a minimum you should consider having the following high level cloud use or ownership strategy goals fleshed out;

• What is the required end result of your move to cloud? This might sound like it's easy, but it's not. This is more than money and it's more than improving your DR or eliminating the need to buy hardware. The move to cloud should have well defined links to your enterprise strategy. Unlike many IT infrastructure solutions (Yes, Cloud is IT Infrastructure) cloud has the distinction of being a technology that could actually change the way your enterprise does business.
  • Does it accelerate your ability to reach new customers
  • Can it improve your ability to expand into new areas
  • Does it allow you to do business with a new class of customer (I.e., you can play with the big kids now)
  • Can it reduce your cost of doing business
  • Does it enable a completely new level of enterprise agility and customer satisfaction
• How will the implementation of your cloud strategy affect your IT group? At a minimum you're going to need to do some retraining, and if you don't plan you might find yourself in a position to have to replace staff. I'm a firm believer that making your team a part of the future will only strengthen your hand and lower the risk you're carrying. There's nothing wrong with investing a little energy in the "Leadership" portion of your management job so you can build team and staff loyalty. Some basic thoughts on the potential change to the organization:
  • Dramatically reduced requirement for
    • sys admins
    • hardware support skills
    • data center ops staff
  • New or Elevated skills will be needed
    • Architecture in the application stack
    • Performance & Reliability tuning and reporting
    • Security
    • Policy development and management
    • Business liaison/engagement managers
    • Contract management
    • Vendor Management and hardware selection strategies
• The good news is that with proper planning your groups natural growth and attrition will mean that if you begin your team transformation now, in three years you'll have the team you need and won't likely need to reduce head count.

What do I mean when I say "Ownership"? I use the term regularly when I talk about data centers. My definition doesn't involve actually physically owning a data center structure, but rather a strategy for understanding and being responsible for all things related to your data center. In other words, having an ownership strategy that's aligned with thinking of the data center as a system ala the Data Center Pulse "Stack". In the case of Cloud "Ownership" I mean the same thing. In order to ensure you have the necessary skills, budget, and vision in place for your cloud, you need to see it as a system and understand what it means to own that system.

Are Multiple Clouds In Your Foreseen Future?

Going forward I strongly believe that most organizations will have multiple clouds, probably greater than three. These clouds will be a combination SaaS, public (EC2, Terramark, etc) and private (vmware, Citrix, etc). As you consider what the cloud enables (agility, flexibility, commoditized hardware, no vendor lock in, etc., etc.) you will start to see that if you want to leverage these benefits you can't fully capitalize on them if you're only using one. Also, the best of breed approach becomes a reality, without the issues of lock in.  

So, if you're going to have multiple clouds how are you going to manage them? Great question, I'm glad you asked.

The last thing you'll want to do is buy/build or contract your way into multiple clouds and not have a strategy for how you'll manage their use and support. This is where cloud Orchestration comes in. The ability to spin up a stack of vm's that fit an infrastructure profile (websphere, .Net, Java) at a moments notice, regardless of the cloud you're requesting it from is key. It's key not just because it's easier, but because it will help you ensure that your governance and enterprise policies are always applied in a consistent fashion. This orchestration and governance also dramatically improves your ability to provide self service to your customers, regardless of whether those customers are IT staff or end-users.

Seeing as how cloud is still a relatively new concept, you can bet there are few solutions in the cloud management & orchestration arena. I believe that ServiceMesh is the only company on the market today that is capable of enabling the opportunities and capabilities discussed in this blog. I will make the disclosure that I've recently taken a position with ServiceMesh, so I'm naturally biased. However, that doesn't change the fact that the statement is true.

How are you going to capitalize on this oportunity?

The future of the enterprise is about to change forever. While many large tech vendors are looking to find ways to lock in their customers, the pursuit of the right to choose is a "given" when it comes to cloud, or at least it should be. The vendors that are most successful in supplying cloud solutions or IT infrastructure for the enterprise will be those companies that adjust their approach and find ways to offer technologies that are measured against new factors like, energy consumed to produced compute, and ease of acquisition with fail in place options.

It's a bold new future and we're on the cusp of being able to realize some incredible new benefits through this revolution in the use of technology. Strap in and hold on, because change is coming. You can be a part of it, or you can be buried by it, the choice will be yours. If you choose to be a part of it, proper planning and execution of strategy can mean your risks can be dramatically reduced and your opportunities improved. Problems Forseen are Opportunities Created.

Sometimes the best answer to a problem is to hit the reset button, but it should probably be the last answer, not the first.

My cohort Pete Silva attended the 2009 Cloud Computing and Virtualization Conference & Expo and offered up a summary of one of the sessions he enjoyed (‘Cloud Security - It's Nothing New; It Changes Everything!’ (pdf)) in a recent post, “Virtualization is Real”

One of the sessions I enjoyed was ‘Cloud Security - It's Nothing New; It Changes Everything!’ (pdf) from Glenn Brunette, a Distinguished Engineer and Chief Security Architect at Sun Microsystems. 

…

Scale – Today Security administrators deal with 10’s, 100’s, even 1000’s of servers but what happens when potentially tens of thousands of VM’s get spun up and they are not the same as they were an hour ago. Security assessments like Tripwire, while work, inject load and what if those servers are only up for 30 minutes?  How can you be sure what was up and offering content was secure?  One idea he offered was to have servers only live for 30 minutes then drop it and replace.  If someone did compromise the unit, they’d only have a few moments to do anything and then it’s wiped. You can keep the logs but just replace the instance.  Or, use an Open Source equivalent every other time you load, so crooks can’t get a good feel for baseline system.

The “scale” we’re talking about is a combination of scaling processes and systems. We don’t often talk about the impact of large-scale environments on processes but security processes are almost always the hardest hit as an environment grows because of the sheer volume of data and systems involved. That said, Glenn’s idea to only allow servers to “live” for 30 minutes is an interesting one, and I am going back and forth between “that’s a good idea’ and “that’s a bad idea” and “there’s got to be a better way.”

One of the reasons this is a good idea is because virtualization provides a snap-shot in time, a known state, a known security posture for the applications deployed within the virtual container. By releasing it and launching it anew, you are assured of the security of the application and environment because it you essentially go back to the beginning. Any changes to the system since the last “launch” are effectively wiped out (logging to an external storage system would be a requirement, of course) and any back-doors, trojans, malware, or rootkits dropped onto the system would be gone.

That would frustrate the heck out of an attacker, wouldn’t it?

But it would also likely frustrate the heck out of end-users who might have been using the application at the time it was released.

There are a couple reasons this is just a bad idea, and the impact on availability to end-users is just the most obvious one. In a live environment it’s never a good idea to just “bring down” an instance of an application – virtual or traditional – that users might be accessing. Doing so severs their connections and wipes out any session state that might have been stored on the server and forces them to “start again”. That said, if you knew this part of your security strategy you could ensure that developers understood this behavior so that the implemented a database-based shared-session model for the applications. If session data is stored in a shared database – on a separate instance – then the potential damage to user sessions is mitigated because it does not rely on any given application instance.

Assuming this is the case, you then have to be concerned about the loss of the connection to the application for users. Again, if you knew this was going to be one of your security techniques then you’d best let the network or application delivery network folks know ahead of time as they can ensure that users are seamlessly redirected to new (or other existing) instances as soon as the one they were connected to is released. Basically you’d have to ensure you had a load balancing solution in place to ensure reliability of access to the application.

This also means it’s more likely you should always have two instances of the application available, and rotating through this up-down-up-down schedule on different time intervals.

Overall you’re likely to incur higher costs with this kind of a strategy as well. It is typical for providers to charge “by the hour” and any partial hour is counted as a full hour. Rotating server/application instances every half-hour would likely incur charges for two instances per hour instead of one anyway. 

This strategy also does very little to address the most pressing security threat facing applications today: tainted user data. That’s going to hit the database, and unfortunately Glenn’s “go back to the beginning” approach to security would be disastrous when applied to virtual environments in which a database is running. You want them to change, to grow, to be modified. It is in their nature to store data and change over time.

So you can’t use this concept for a virtualized environment in which a database is deployed. It would be detrimental to the health of the business.

But there’s something to Glenn’s idea that’s certainly appealing when part of a broader security strategy. What his “up-down-up” technique is designed to prevent is compromise of the system, i.e. trojans, worms, viruses, and malware inserted into the system that can be used for illegitimate access or as part of a larger botnet. HIs technique certainly addresses those security risks by effectively wiping them out on a regular basis. What’s not accounted for is the injection of malicious code into the database, which cannot be so easily “reset.”

Perhaps this is a job for Infrastructure 2.0?

If we employ the use of an infrastructure 2.0 capable application delivery network we can utilize Glenn’s technique in conjunction with other security technology to provide better coverage in a more dynamic way. Consider that the integrated network and application network security capabilities of the application delivery network can protect application instances against web application attacks, especially those that are really targeting the database, e.g. SQL injection.

Also consider that an application delivery solution can provide the failover capabilities required to assure availability in an environment in which instances may be going down and coming up in a highly volatile pattern.

That addresses the “bad” and the “ugly” impact on end-users resulting from Glenn’s “up-down-up” technique, leaving us only with the “good”. 

But it really doesn’t address the root of the problem, the reason Glenn suggests going back to the beginning in the first place: volatility and change. Scaling security processes across thousands of virtual instances is problematic, I agree, but one of the reasons it’s so hard to scale is that you don’t know what’s going on. There’s currently no real collaboration across the entire infrastructure. Security folks can’t get a good feel for what’s going on in a large scale, dynamic environment because the information they need to correlate and assess the current security posture of the environment and applications is dispersed across the infrastructure.

What’s needed is an overarching system that can integrate security solutions with the rest of the infrastructure. When a virtual environment is brought on line the security infrastructure needs to know about it –not just to apply the proper policies but also to assess its current posture and ensure it is added to the pool of resources that needs to participate in the larger security scheme. If a HIPS (Host Intrusion Prevention System) is used to monitor a system for intrusion and its alarm is triggered, that information needs to be imparted to the rest of the infrastructure. If a virtual machine is potentially compromised it should be immediately removed from the available pool of resources. That requires collaboration across the entire infrastructure. If part of the launch process includes a vulnerability scan of the application and that scan comes back positive perhaps the instance should not be allowed to launch, and the infrastructure notified immediately so that it can take whatever steps are necessary, such as automatically virtually patching the vulnerability if possible and allowing the instance to launch while notifying security and developers that there’s a vulnerability in need of patching.

Cloud computing and virtualization are going to force integration and collaboration into the fore of architecture design necessarily. The scale of systems using virtualization is growing and becoming less and less manually manageable, which will inevitably result in more automation and orchestration at the infrastructure layer.

Let’s not forget the myriad pieces of security software that provide valuable information and threat mitigation are also part of the “infrastructure 2.0” family, as it were. We need to start thinking more broadly, more strategically about how to leverage collaboration across the disparate functional silos within IT to come up with better solutions to address security and its associated scaling challenges in a cloud computing environment.