Introducing a Metadata system into the cloud

Introducing a Metadata system into the cloud
Organizing & structuring massive amounts of cloud data

One of the most pertinent problems cloud computing issues that we (as users and enthusiasts) are facing is the congregation of massive amounts of data. The problem is that clouds across the globe are beginning to require ever increasing amounts of additional storage, this comes on the heels of recent findings that given the current rate of growth, cloud computing will not be able to utilize the current storage methods to meet the growth of the demand. In other words, we need to begin instituting a new way of organizing data for storage that makes better use of our resources (current and future). The truth is that a comprehensive solution to storage problems as well as things like gaps in data has not been properly addressed. These issues along with several others directly related to data storage and accumulation are not being dealt with; this has the potential to seriously downgrade the authority and effectiveness of cloud computing over the long term.

The simplest most obvious solution that has been proposed is the adoption of a metadata system for all clouds (and perhaps even subsystems).

What is a metadata system?
The most relevant and simple example of a metadata system is that of a library card catalogue. What this does is give you a quick overview of the item you're looking for, including a complete description of said data, as well as other relevant info. A metadata system would essentially fulfill two crucial tasks for use in the cloud.

1. Create a streamlined method for organizing and accessing all data in the cloud; speeding up user ability to get from one place to the next.
2. Provide cloud managers (or those in charge) with an exact catalogue of useful / relevant / non-redundant data.

Creating a list that basically identifies all data on a cloud will allow you to both eliminate redundancies as well as restructure the way a cloud is set up. This would also allow you to maximize storage potential and restructure certain elements to increase performance and availability.

Any type of system that houses or provides access to a vast array of records / consumer data for example, already uses a metadata system. Businesses, medical organizations and retailers often take advantage of some type of metadata system in order to offer quick and easy access to the large volumes of data that they often store. But we're not talking about using one of the stock in trade metadata systems to help restructure a cloud here; the new type of metadata system will be given free reign over the entire cloud. There are already some automated cloud components (like lobots or cloud management software) that trawl over cloud data looking for indiscretions and errors; a metadata system needs to function in a similar way as these components.

There are some serious roadblocks to enacting such a system, however. For one, all applications, OS's, storage centers and virtualized processes (among other things) must be reconfigured to interact with and make use of a metadata system. As you might expect such a thing would be a massive undertaking, very difficult to realize (but not impossible). Then, a system or series of tools would need to be developed in order to either eliminate superfluous data or resign it to another holding area where it could be further evaluated (either automatically or manually).

Why is 'data overload' a concern among those in the cloud?
Just imagine you have a computing system, everything is running fine, it is expanding and growing at an acceptable rate; everything is fine. Then, out of the blue, the rate of growth and data expansion begins to rapidly accelerate. It suddenly becomes necessary to add new storage hardware on a continual basis, which is expensive. In the midst of this, you begin to notice that as the total amount of storage space required increases, you are accumulating extremely large amounts of 'junk data' as well. So the dilemma is thus; if a majority of the data you are paying to store is utterly useless, why not eliminate the junk data and save some money?

One of the selling points of cloud computing is its effective and efficient use of both energy and resources, if we allow it to become bloated and overrun by terabytes of crap data, what's the point of it all? In order to remain relevant and competitive, we must deal with this issue (data overload) head on. Cloud computing must not lose its elasticity if it is to replace the current standard.