EU Personal Data: Safe Harbor vs Home Port

As you probably know, Safe Harbor acting for European personal data in the US from 2000 was recently ruled out by the ECJ (European Court of Justice) to be insufficient. This changes a lot the background for cloud computing consumers in Europe.

OK, what is this all about bit by bit:

1. Europe has its own privacy laws. This is related to the European citizens personal data protection and is regulated by the Data Protection Directive from 1995.
2. There are a lot of transnational US businesses that are storing, aggregating, analyzing the global customer data in the US-based datacenters. Such businesses can be of infrastructure level (like cloud services providers, hosting services, etc.), online services (social networks, blog platforms, search engines, etc.), e-commerce players and so on.
3. Safe Harbor Privacy Principles were developed starting from 1998 and enacted in 2000 to make it possible for the European personal data to travel transatlantic and be handled there in safe manner.
4. Last several years revelations of NSA activities and USA Patriot Act enforcement that are bypassing the European personal data protection and privacy laws.
5. Safe Harbor is not sufficient to protect European citizens personal data anymore, as ruled by the court in 2015.

What will be the most likely consequences? Will it help European CSPs to rise and gain the market share? Will this create the workplaces in Europe? How will this boost the cloud consulting companies?

What we can see to the moment, the big US CSPs are opening more datacenters to keep the European data (and metadata) in Europe:

• Microsoft announced the plans for UK expansion in 2016 (Azure, Office 365 services datacenter): http://www.windowscentral.com/microsoft-plans-open-new-data-centers-uk
• Amazon announced the new cloud datacenter to open in UK in 2017: http://fortune.com/2015/11/06/amazon-web-services-uk-data-center/

One one hand, cloud infrastructure business is a mass market with it's low margin economy, it is only possible to compete there having the global scope and huge resources. So probably Safe Harbor strike down will not significantly help any new European players to benefit from this situation, however new datacenters to be open in Europe should add more workplaces in EU member countries.

One the other hand, Europe has it's own strategy for the cloud computing (https://ec.europa.eu/digital-agenda/en/european-cloud-computing-strategy), C4E (Cloud-for-Europe) initiative, ECP (European Cloud Partnership, https://ec.europa.eu/digital-agenda/en/european-cloud-partnership) organization, so why not to coordinate/implement something of a level of pan-european CSP?

May the Cloud Force be with you. What the recent movie tickets services crash teaches us.

Have you heard how frustrating was ordering tickets for the next Star Wars: The Force Awakens? Big online services like Fandango, MovieTickets, AMC, Regal, Cinemark and others across the globe were crashing when the fans flooded them in attempts to book the tickets just after the announcement.

“Cloud Force” could really help here. As this was a significant peak in booking service consumption, this could be addressed perfectly by the the cloud:

• Rapid elasticity would allow to handle a sharply increased number of consumers without noticeable degradation of the service level. The computational nodes could be added automatically and transparently and deprovisioned when not needed any more.
• Hybrid cloud scenario would allow to borrow the required computational power from the public cloud without any need in investing in dedicated infrastructure.

Even if this peak would be unexpected, the cloud could handle it based on the utilization metrics; however in this case the spike was perfectly foreseen so the anticipated load could be addressed with schedule-based elasticity triggers or even with manual provisioning.

The automatic elasticity comes extremely smooth if you deal with the cloud on the PaaS (Platform-as-a-Service) level, everything will be handled for you mostly transparently. If you want to keep the resources under control on IaaS (Infrastructure-as-a-Service) level, for some of the most popular public cloud providers those features that enable automatic elasticity would be:

• Amazon Web Services: Auto Scaling, Elastic Load Balancing, Auto Scaling Group
• Microsoft Azure: Cloud Services, Azure Load Balancer

Basically, the cloud wouldn't only help to handle such peaks in usage and thrive, it would also make it happen in really cost-effective way, without any requirement of statically assigned mostly idling infrastructure.

This is the news coverage:

http://www.wired.com/2015/10/star-wars-online-ticket-sales/
http://www.cnbc.com/2015/10/19/star-wars-ticket-demand-crashes-uk-cinema-websites.html
http://money.cnn.com/2015/10/19/media/star-wars-movie-tickets/
http://www.forbes.com/sites/hayleycuccinello/2015/10/20/star-wars-presales-crash-ticketing-sites-sets-record-for-fandango/

Those are apologies from some of the services:

https://twitter.com/ritzycinema/status/656385731687030785
https://twitter.com/ODEONCinemas/status/656041802013810688

Cost Management for Projects That Use Public Cloud

Project cost management 

Cost of the project is one of the biggest aspects of the management. This is a typically achieved through the set of well-established approaches. According to PMI, the processes that help with project cost management are:

1. Resource Planning
2. Cost Estimating
3. Cost Budgeting
4. Cost Control

Let's consider how public cloud associated costs can be addressed withing this framework (cloud costs will be only the part of all project costs, yet we will focus on them specifically)

Public cloud resource planning 

Public cloud can be considered as a set of resources which depend on the cloud service model: PaaS (Platform-as-a-Service), IaaS (Infrastucture-as-a-Model), or SaaS (Software-as-a-Service). More specifically those resources would be:

1. IaaS: server instances usage, storage space, network bandwidth, inbound and outbound traffic, load balancing, etc. 
2. PaaS: developers accounts, technologies/services/frameworks usage, etc.
3. SaaS: user account, applications/services usage, etc. 

Based on the type of the cloud, resource requirements should be defined.

Public cloud cost estimating 

As soon as we have resource requirements defined, we can use our WBS (Work Breakdown Structure), activity duration estimation and resource rates to make the estimation for the cloud related costs of the project.

Resource rates are provided by the CSP (Cloud Services Provider) and sometimes the costs structure is not very easy to grasp. For example, for AWS (Amazon Web Services), EC2 it looks like this. You have to decide about a lot of things to be able to calculate the costs, most likely this will require involvement of the project's System Architect or Tech. Lead. (There are also some online calculators, yet the process is not that simple)

Public cloud cost budgeting 

After the estimation of the required resources is done we can use it with WBS and project schedule to come up with the cost baseline.

Cost baseline for the whole project will include different costs, we are considering only cloud-related costs here.

As the baseline goes over time, it will go through the different phases of the project, each phase will require its own specific amount of the cloud resources. It would be pretty typical to expect that in the beginning of the project the project team will require the cloud resources for prototyping and closer to the project end a lot of testing will be performed in the cloud. Speaking of the testing activities in the cloud, makes sense to mention that load/scalability/performance testing can be most resource consuming and hence it can require significant part of the whole project cloud budget.

Public cloud cost control

This part is of the whole cost management processes set can look as the most fitting to the current public clouds state of the art. As one of the main cloud's features is "Measured Service" (see NIST cloud definition I mentioned in my previous post Cloud-related projects: when your backend is really based on cloud services?) you are usually in full control of the cloud costs to the moment. This means that you can use very structured reports to see how the cloud spendings are attributed. This can help a lot to revise the estimates, make needed updates or corrective actions. This part is where the public cloud is typically shining.

Cloud-related projects: when your backend is really based on cloud services?

A person in charge of the project leadership and finally project success should understand the meaning of cloud-based development project, as this is a really big point (although this game changer is already mature enough).

Cloud-as-a-Buzzword

Due to the big hype for the last 5-7 years, we encounter the word "cloud" and projects related to "cloud services" development very often. In fact, big deal of such cloud-based projects are not about cloud at all. 

Sometimes backend is called "cloud services" just for the marketing purposes (yes, cloud is still cool), but this can be also a lack of understanding of what the "real cloud" means. You can meet a lot of developers (even experienced enough) that can say something like "cloud is actually nothing really different from the traditional client/server with backend in some virtualized infrastrucure (or even just Internet)". It is not like this, however.

When you really develop cloud services and the project is about cloud?

First of all, to be clear, by the "cloud services" I mean not the services CSP (Cloud Service Provider, like Amazon Web Services) provides to the consumers, but the services that represent the back-end of the system which really benefits from the cloud-based architecture. 

Does this mean that any services, being deployed to the public cloud (AWS, Microsoft Azure, etc.) or private cloud (based on OpenStack, etc.) are automatically becoming cloud services? Well, no. Only those services that can really use the cloud's nature are cloud services.

What is the cloud? I think NIST (National Institute of Standards and Technology) gives very good definition for this type of the infrastructure. According to this widely accepted definition. these are the essential characteristics of the cloud infrastructure (with my explanations):

1. On-demand self-service. This means the cloud consumer should be able to perform all cloud provisioning actions without any help from cloud provider. This can be fully automated with different APIs or done manually through "service catalogue" (usually Web-based user interface).
2. Broad network access. Stays for ability to consume the software services in the cloud using different types of clients through standard platform-independent protocols (it is not about broadband connection).
3. Resource pooling. This is related to the fact that cloud infrastructure is shared among different "tenants" to optimize the utilization. This makes it efficient and usually is a big supporting point for the switch to the cloud infrastructure.
4. Rapid elasticity. Truly great ability of the cloud to quickly grow the amount of provisioned resources for "tenants" when needed and to contract them when not needed any more.
5. Measured service. This means in the cloud all resources are "commodities" that are standardized and consumption is measured (like electricity consumption).

Most significant enablers for the services developers there are "rapid elasticity", "broad network access" and "on-demand self-service", to my mind.

OK, how to use this knowledge?

It is important that the cloud is not just virtualized infrastructure, not just virtual machines, etc. This is highly automated, elastic, controllable, efficient environment that should be employed in a right way to benefit from it.

You'd better ask your System Architect, Lead Developer/Expert or other person in charge of the technical decisions in your team how your backend services are compatible with cloud-deployment scenarios, how they address scaling in the cloud, what are the interfaces to them that enable services consumption, how is cloud-backed automation levereged, and so on. 

Cloud services can make a real difference for the system/product you develop (especially in terms of scalability and automation) if this is really about using the features of the cloud infrastructure.