Next week, I am once again heading back to the IBC show in Amsterdam, considered by some to be the world’s most influential media, entertainment and technology show. For me, a large part of the IBC experience is created by the unique venue of the event (the RAI), the exceptional quality and brilliant diversity of the delegates and the outstanding city of Amsterdam, filled with splendid restaurants and bars.
Slated for 13–17 September 2019, this year’s IBC has 400 inspirational speakers and boasts a world-class exhibition, packed with 1,700 exhibitors (including our Caringo team at Stand 5.C33). This provides a perfect environment in which to network, build relationships and discover the latest trends and technologies. I always find the sense of opportunity overwhelming, and this year, I am excited to discuss with attendees the latest developments at Caringo.Expanding Caringo’s Global Footprint
Since last year’s IBC, Caringo has greatly expanded its global footprint with landmark deployments in Europe, the US and East Asia. In the UK (where I reside), Swarm emerged as the leader after object storage performance benchmarking at the Science and Technology Facilities Council’s (STFC) JASMIN super data cluster. (Download the STFC Object Storage Benchmarking Case Study & Whitepaper to learn more.)Addressing More M&E Use Cases
As our pure object storage platform has been adopted by organizations and used for new use cases, the needs and input of our customers has helped to shape the evolution of our Swarm pure object storage platform. With our expansion of media & entertainment (M&E) use cases and customers, we have sharpened our capabilities to integrate into digital video workflows.
Over the past year, Caringo Swarm has been adopted by elite streaming services and implemented as an on-demand archive for professional sports teams.Visit with Caringo at IBC2019
Join us at Stand 5.C33 in the IBC exposition to chat with me and our other storage experts. We will share with you how our upcoming Swarm 11 release will revolutionise your workflows and help you collaborate and scale your content with speed and simplicity. If you would like to obtain our code for a complimentary IBC expo pass or to make an appointment with us at the event, contact us.Learn More About Caringo Swarm Object Storage
Can’t make it to IBC this year? Join VP Marketing Adrian “AJ” Herrera and Director of Product Eric Dey for a look at Accelerating On-Demand Access to Video Archives with Swarm 11. They will show you what’s new in Swarm 11 with a live demonstration of our newest features and interactive Q&A.
How valuable is your time?
If you are in the M&E industry (or deal with digital video), what is the value of being able to complete more projects, send videos to employees/editors faster or stream more videos to subscribers?
Once organizations grow to the level where their workflows are being strained, they often look to remedy that starting with the front-end application. Once they get to the storage layer, they are usually looking for the most economical storage to satisfy the requirements of the front-end application. This often boils down examination of storage to a financial ($/GB) rather than a functional analysis. The problem with this approach is that the functional value added by object-based storage platforms like Caringo Swarm gets lost.
For object storage specifically, the time and resource optimization value gets lost. I often hear object storage positioned as cheap and deep storage, which I think is a misnomer.How Does Swarm Object Storage Go Beyond Cheap and Deep?
Caringo Swarm object-based data storage is scalable and strategic for asset access. That is, it provides extensive benefit strategically by serving up effortless scale for rapidly growing archives of data and granting instant access to those assets.Wait, What About Tape Storage?
While tape will always be cheaper and deeper and is perfectly adequate for files and assets you may not ever retrieve, it lacks one of the most beneficial features that you get with an object storage platform like Swarm: on-demand access.Scalable Object Storage
We have a lot of excellent resources that go into detail on how scalable object storage is. Two that I recommend are:
- What are the 5 Tiers of Storage for New Video Production Workflows to get a comparison of the various storage technologies available
- Object Storage Performance Benchmarking for a detailed overview of performance characteristics of object storage at scale
Now, let’s go into how object storage has evolved as a strategic tool for asset access.Archive, Active Archive and On-Demand Archive
When you look up the definition of archive, you will most likely find it associated with the word “preservation.” Historically, archive has been a backup or a copy of an asset made for when and if that asset is needed again. In the storage world, it has long been clear that there is a large portion of digital assets that need to be continually accessed, leading to the term “active archive.” The word active denotes access but doesn’t specify time to access.
After talking to many in the M&E and Digital Video Industry, it is clear that time is their primary issue. Specifically, the time to move video from remote locations into their network, the time it takes to move working files to different locations, the time it takes to restore files for reuse from archives and the time it takes to efficiently stream content to internal or external viewers. This is why at Caringo you will often hear us use the term “on-demand archive.” On-demand denotes immediate access. You don’t need to wait for a video to be found, a tape to load, or wait for the mail to arrive with the thumb drive or hard drive. Your archived video is always instantly accessible to your applications, internal employees and (if you authorize access) to the public for on-demand viewing.How Does Swarm Object Storage Save You Time?
In addition to providing immediate access to archived video, Swarm object storage also enables multi-tenancy, content and metadata management with integrated search. All of these features save both your administrators and end-users time by providing an intuitive web-based platform where they can have direct access to their portion of storage (private cloud storage). They are able to search the assets they are authorized to view and can also update assets by adding relevant metadata.How do You Measure the Value of Time in Your Organization?
Where the rubber meets the road is how you measure or assign a value to saving time. What if you could complete 10–20% more projects per month? What if you could spend 50–60% less time managing access to content across your organization or with clients? How about if you could provide perpetual access to a specific portion of your archived video for a new VOD (video-on-demand) service? However you place value on time, I can tell you this…it’s not by looking at a $/GB calculation!Swarm 11, Focused on Optimizing Time
If you are interested in learning more about how Swarm object storage optimizes your time, look no further then our next webinar. Eric Dey, Director of Product Management, and I will discuss Accelerating On-demand Access to Video Archives with Swarm 11.
And as always, we have a team of experts at Caringo that can help you figure out how to measure the value of plugging in an on-demand archive like Swarm into your workflows. Contact us if you would like to setup an overview.
The post How Object Storage Optimizes Your Time & Resources appeared first on Caringo.Related posts:
I travel a lot for work. July alone, I was on a plane every week. New York, Austin, DC and Chicago were a few of the stops I made to visit great customers and partners like CatDV, Masstech, Diversified, AWS and Cinesys. And with each negotiation through an airport, I witnessed the same struggle with each passenger…
control of their content.The Balance Between Content and Storage Space
For some travelers, it was an epic battle to see if they could get their bag into the ever-shrinking overhead compartments on airplanes. Overwhelmed airline attendants do their best to help, but they have to juggle the enforcement of the rules (size of bags, # of bags, etc.) and customer happiness. Add in the fact that some customers get preferential treatment because of status (first class, frequent traveler, cost paid for a seat, and so on).
Watching this scenario play out reminded me of the struggles many organizations face every day with storing data. I know, I know…how do flight attendants and passengers and luggage battles remind me of data storage, and specifically, Caringo’s award-winning Swarm Object-Based Data Storage?
Please allow me to explain. But I have to warn you: grab your tissues now because there could be tears.What is Happening in the Content World Today?
Files are getting larger, people are deleting less and access is happening over more devices than ever before—these are three things I think we can all agree are happening in the content world today.
As resolution increases the resulting files are jumping in size, more cameras are being added to the world, and mobile devices are becoming more powerful (and 5G supported), making the dynamics of storing and managing content harder than ever.How Can Swarm Object Storage Help with Storing Data?
Swarm provides a platform that allows users to store all content and to easily search for exact items using traditional means (file name, date, location) or more modern ways (that is, metadata search). Swarm also allows you to retrieve the exact data you need. This is similar to the well-organizer traveler who has the items they need most in the seat pocket in front of them.
If you think of the flight attendant as the administrator of the plane—making sure only the correct passengers enter the airplane and ensuring each passenger only uses their allotted space but in a self-service manner. That enables passengers use the system in an organic way that fits their travel flow.
Swarm does all these same things. Simply put, Swarm creates an easy-to-manage storage platform that allows content to be efficiently accessed. With powerful built-in metadata search, it’s easy to find whatever it is that you need in the object store. You can restore a whole clip or just a few frames as you need to. Best of all, Caringo Swarm slides content right into your existing workflow. And, given that it is less expensive than public cloud and more reliable than tape, Swarm Object Storage will be the smartest investment in your environment.Buckle Up for IBC2019!
This September, I’ll be heading to IBC2019 at the RAI Amsterdam 13 – 17 September. IBC (originally the International Broadcasting Convention) has evolved from its technical broadcast roots to encompass the entire breadth of media creation management and delivery. Visit us at IBC Stand 5.C33 to speak with our object storage experts to hear what’s new in Swarm 11 or register now for our upcoming webinar: Accelerating On-Demand Access to Video Archives with Swarm 11.
The post It’s a Bird! It’s a Plane! It’s Swarm Object Storage! appeared first on Caringo.Related posts:
“The best investment is in the tools of one’s own trade.”Benjamin Franklin
In today’s fast-paced world, the data storage landscape changes at a blindingly fast pace. It seems that every day there is a new start-up claiming they have “invented a revolutionary new way to store and manage your data.” But, when it comes down to it, many are just re-inventions building on an existing archetype (that is, someone else’s ideas with a bit of a “twist”).What are the Different Types of Storage?
There are three primary ways to store data. File, Block and Object storage. For more information on this topic, read our Back-to-Basics blog.
Once you understand the different types of storage, it may become obvious which type of storage you need to add to your existing infrastructure or deploy for your use case.How Does Object Storage Fit Into Your IT Infrastructure?
Often, object storage is used to augment your existing storage, whether that be SAN or NAS. In our on-demand webinar How Does Object Storage Fit into Your IT Infrastructure?, our CEO Tony Barbagallo and Sr. Consultant John Bell take on this topic. They review common use cases for object storage and talk about how it fits into your existing storage network.Should I use Object Storage Instead of SAN, NAS or Tape Storage?
Object storage is ideal when content will be accessed again or reused, especially over the web (e.g., digital video or digital evidence). Object storage is also useful when you need a cold archive but want the data to be more secure and simpler than to retrieve than tape.
However, if you are running a highly transactional workload, you will want to use SAN or NAS as your primary (or tier 1) storage.
The type of storage you need varies by use cases. Data storage is never a one type and size fits all proposition (and beware if a vendor tells you that their data storage is a good fit for every scenario).
In the chart below, we show which type of storage suits various use cases.5 Signs You Need Object Storage
In my blog NAS vs. Object 2019: If Not Now, When?, I outlined the telltale signs that you need to add Object Storage into your existing infrastructure. Here is a high-level recap:
- You have lots of unstructured data to store that may be accessed again
- Your primary storage devices are overloaded and slow
- It is difficult to locate data
- You want to intelligently mine your data for insights
- Your IT budget is tight and your storage needs are high
In the data storage industry, tools can be quite nuanced. So, understanding the twists (aka, the features and functionality) of various products is critical in making your product selection. Here’s just a few of the things that set Caringo Swarm apart:
Stability: Having launched our first product in 2006, our Swarm Object Storage is field-hardened and highly reliable.
Scalability: Scaling from small to 100s of Petabytes is one of the hallmarks of our pure object storage approach as well as a testament to our long-standing relationships with some of our earliest customers.
Metadata: By storing metadata with the object rather than in a separate database (a method employed by most object storage vendors), we unleash the full potential of metadata. To learn more, hear what Ryan Meek, our Chief Solutions Architect has to say on the topic in our Using Metadata with Object Storage webinar.
Support: Our Support team has some of the most experienced storage engineers in the business and with a global presence, help is never far away.
Resilience & Recovery: Undoubtedly, this is one of the most important factors in choosing a data storage solution. Register now to attend our August 20 Tech Tuesday webinar live as VP of Engineering T.W. Cook joins Sr. Consultant John Bell to review just how Swarm keeps your data safe and recovers it when needed. They will be taking questions throughout the broadcast.
(Note: All of our webinars are recorded and available on demand after the broadcast concludes.)How Can Caringo Help?
“There is a great satisfaction in building good tools for other people to use.”Freeman Dyson
We love what we do because we see the benefit that our Caringo Swarm Object Storage and accompanying products such as SwarmNFS, FileFly, Caringo Drive and Single Server bring to organizations. We know that not every organization has a storage architect on staff, so at Caringo, we have a number of highly qualified storage architects, consultants and engineers that are happy to help you. Just visit our contact page.
The post Selecting the Right Data Storage Tools for the Job appeared first on Caringo.Related posts:
I was recently at the SVG (Sports Video Group) Content Management Summit in New York City where our CEO Tony Barbagallo was on a panel titled Object Storage for M&E: Making it Work for Broadcasters On-Prem and in the Cloud. Right before the panel started, one of the attendees asked the person sitting next to him, “What is object storage?” The first person who answered worked for a large file transfer company. He defined object storage as a type of storage target that a data manager can write to. Once he finished, I interjected with a brief description of the technical differences between file, block and object storage.What is Object Storage?
After seeing the expression on the face of the attendee who originally asked the question, I realized that I had also missed the mark. So, I launched into explaining object storage using the popular analogy of a valet parking system. In this analogy, the content is your car and all you need is the ticket to retrieve that content. While that explanation resonated with him, there was still clearly a disconnect between the technical aspect of easily retrieving a file and the benefits of using object storage.Perspective is Everything
Three things became obvious:
- Whoever is defining a technology will do so from their own perspective.
- The person looking for the definition will process the definition based on their own experiences.
- We need a way to describe the actual benefit of object storage to the end user.
Armed with these revelations, it became clear that I needed to come up with a new analogy for object storage that explained the benefits of the technology from the end-users’ perspective (e.g., a subscriber of a video streaming service or a viewer of sports video), rather than from the perspective of a Storage Infrastructure Manager or IT Executive.
In my quest to come up with a new analogy to illustrate the benefits of using object storage, I thought about the process of packing for a trip. For people who travel extensively, luggage can become an obsession and packing an art. After all, the success of your trip depends on having the right items with you, just as the success of your business endeavor relies on being able to access or stream content rapidly.
There are a number of tricks to packing quickly and efficiently as you need to grab the correct items for each trip. After all, what is a beach getaway without casual clothes, swimsuit, sunscreen and comfortable footwear? Or a business trip without proper attire and grooming products?
Think of your suitcase as the video you plan to view, whether it is from a PC, mobile device or connected to a device such as Roku, Fire TV Stick or Apple TV). The combination of the items in your suitcase are like the bits of data needed to display a video file. With object storage, your suitcase is always packed and flight ready, so your video is instantly available for delivery over the web.
Whereas object storage resembles a pre-packed, flight-ready suitcase, file storage requires that a suitcase be packed every time that video is requested.
Imagine that the items you need to pack in your suitcase are strewn around your house and you must go fetch each and every one prior to packing. This is similar to how file storage works. With file storage, the bits and pieces you need for the video are spread across a storage device, and they must first be assembled. Then, the assembled file needs to be sent to a web server to deliver over the web.Object Storage Accelerating Archive Access and Agility
Streamlined video access is just one of the many benefits of pure object storage that we have been working on for over a decade. And we are not done. On September 10 we will be launching Swarm 11, which is focused on accelerating archive access and agility. Our partners and customers can request a preview before launch. To do so, just reach out to your Caringo Representative. If you are not an existing customer or partner, make sure to sign up for our monthly Caringo Buzz newsletter and we will send you an invitation to one of our upcoming launch webinars.
The post Object Storage Delivers Archive Access and Agility for M&E Workflows appeared first on Caringo.Related posts:
Data recovery in Swarm works hand-in-hand with its data resiliency features. In our last blog post, we discussed how Swarm distributes and protects data from the possibility of failure. When a failure actually happens, Swarm performs active recovery with the goal of again achieving full protection for its data. We regularly perform a kind of “whack-a-mole” test with up to 14 sequential failures. A properly configured cluster can have multiple simultaneous hardware failures and achieve full replication again in minutes.What Happens When a Storage Volume Fails?
Of all the types of failures, volume failures are perhaps the most common. Even with annualized volume failure rates of a low 2%, a large Swarm cluster (e.g., PB+) can potentially have one or more failures a month. Because of Swarm’s parallel active recovery mechanisms, a larger cluster will also recover from such a failure faster than will a smaller cluster. Few storage solutions in the market scale its recovery mechanisms in this way. Swarm’s active recovery means data loss only happens with true catastrophic failures or gross neglect.How Does Swarm Recover?
Volumes can be physically moved from one chassis to another, either newly introduced or already running in the cluster. The data on the volume will then be available for use after mount. While Swarm can recover from the permanent loss of a chassis (along with its volumes), chassis loss is usually a temporary problem remedied by a new power supply or network card.
Performing full recovery in a chassis or subcluster loss may be counterproductive if the data is known to be on viable disks. In such a situation, Swarm can alert to this condition and accept administrator guidance on whether full recovery of all volumes should be performed. In the absence of this feedback, Swarm will proceed with the recovery. As described in part 1 of this blog, Swarm will be able to read and write all cluster data in either of these temporary loss scenarios.Best Practices for Cluster Configuration
Some Caringo customers keep multiple clusters for high availability and failover. Clusters can be easily configured for remote replication and multiple clusters can be configured to mirror each other. With small network configuration changes, either cluster can be the primary access for an application or user base or load balancing to both clusters can be used to serve out exactly the same objects. Should one of those clusters suffer a catastrophic failure, the other cluster can carry the load while the first one is repopulated.Take the Worry Out of Protecting Data
With its combined data resilience and recovery mechanisms, Swarm takes the worry out of protecting your data and making it always available. These mechanisms “just work” with little configuration or manual intervention. This is especially important with remote data centers that may only be periodically serviced.Learn More About Data Resilience & Recovery
Register now for our August 20 Tech Tuesday webinar, Data Resilience & Recovery in Swarm Object Storage. It will feature T.W. Cook, VP Engineering, and John Bell, Sr. Consultant, and include live Q&A throughout the webcast.
The post Resilience & Recovery with Swarm Object Storage, part 2 appeared first on Caringo.Related posts:
Data resiliency has been a feature of Swarm from the beginning. The architecture of Swarm object storage has never had a single point of failure. One or more Swarm nodes may be down for maintenance or due to hardware failures and the rest of the Swarm cluster keeps running, servicing requests.
Since the data in Swarm remains available, users are likely to not even know about the outage. Because a larger Swarm cluster has better throughput than a smaller one, the loss of one or more nodes generally only has a small impact on performance.Why is Swarm Object Storage Resilient?
One of the resiliency features is that Swarm keeps multiple replicas of objects and it will not co-locate replicas where single failures are likely to take them both out. It’s the old idea of “don’t put all your eggs in one basket.” Replica distribution is discussed in my 2016 blog series (Protecting Data in the Cloud Age with Object Storage) as well as in our Protecting Data with Caringo Swarm Object Storage whitepaper.
The level of protection is easily configured, both at the cluster and at the object level. As a result, Swarm and the data it stores are resilient to failures at platter level, the level of the disk drive, the chassis, and even logical subclusters. In the latter case, customers can configure Swarm to know about a cluster’s network topology and location information.How Replication and Distribution Increase Resiliency
Full replication and distribution of replicas (and erasure coded segments) are achieved from the time of the initial object write. Swarm’s health processor is a background task that is continuously checking for the presence and locations of replicas so that a cluster’s data is always protected from all but the most catastrophic failures, such as a fire or flood. Various remote replication options provide data resiliency for those possibilities. You can learn more by reading the Elastic Content Protection Technical Overview paper.Learn More about Resilience & Recovery of Swarm Object Storage
Data resiliency is an integral part of Swarm object storage software and we are proud to have customers with clusters that have been operational for over 10 years, through all manner of hardware failures and upgrades. In next week’s blog post, I’ll discuss the data recovery features that come into play when a failure actually happens.
Register now for our August 20 Tech Tuesday webinar, Data Resilience & Recovery in Swarm Object Storage. It will feature T.W. Cook, VP Engineering, and John Bell, Sr. Consultant, and include live Q&A throughout the webcast.
The post Resilience & Recovery with Swarm Object Storage, part 1 appeared first on Caringo.Related posts:
Dark data. Sounds a bit sinister, doesn’t it? Let’s unravel the mystery of dark data and talk about the complex issues that must be considered and the ramifications of storing or not storing that data for businesses and organizations.What is Dark Data and Should You Bother to Store It?
Wikipedia defines dark data as data “acquired through various computer network operations but not used in any manner to derive insights or for decision making.”
The volume and rate of collecting data can easily exceed the capability of most organizations to properly tag, store and analyse that data. Not surprisingly, given how difficult it can be to identify the wheat from the chaff, it has become a common practice to store all the data that is generated.
With over 2.5 quintillion bytes of data created every single day, and an estimated 1.7MB of data created every second for every person on earth by the next year, this becomes an increasingly pressing issue for storage architects and IT departments.
Storing all that data creates everything from compliance issues to overburdened storage systems and also raises the possibility of ransomware threats, but deleting it has the potential to cause even more problems. What if you accidentally delete something you need or that might prove to be invaluable later on?
So, most organizations continue to add more storage as they accrue data, and much of that data is unstructured data.What is Unstructured Data?
You could say that enabling organizations to cost-effectively store unstructured data is our business. After all, we’ve been doing that since our first product release in 2006.
Unstructured data is quite abundant in today’s IT landscape. It can be just about anything, from music recordings to medical imaging to video footage. The defining characteristic of unstructured data is that it is not stored in a structured, predefined format. That makes it challenging not just to store, but also to manage.How do you Store and Manage Unstructured Data?
Over the years, the Caringo team has helped numerous customers store, organize and access massive amounts of unstructured data with our Swarm Object Storage Software. Check out our case studies for detail about how we helped organizations like the STFC Scientific Computing Department, Texas Tech University and NEP in the Netherlands.When you tier data into Swarm object-based data storage, you benefit from continuous built-in data protection, management, organization and search at massive scale. As the pioneer in object storage technology, Caringo products have some distinct differences in methodology that give our customers a significant advantage. While we cannot cover them all in just one blog, part of the Swarm difference revolves around our integration of Elasticsearch and how we store and use metadata (a.k.a., data about the data).
To learn more, watch our Tech Tuesday webinar about using metadata with object storage on demand or read the summary that follows the webinar.How Does Caringo Use Metadata & Elasticsearch to Illuminate Dark Data?
Metadata and Elasticsearch are the key to making data easy to find in Swarm Object Storage. This is a topic we have addressed in a number of blogs and webinars, including our most recent Tech Tuesday webinar (using elasticsearch with object storage).
Using Swarm’s extensive custom metadata capabilities and Elasticsearch simplifies the task of locating discrete types of data in a large data store. It gives you dynamic organization of content with classification, key words, descriptive content and multiple methods to track content with no separate big data project required.
Once you have data in a Swarm Object Storage cluster, the content of it and the value of it are illuminated, so you can reap insights and potentially realize new ways to monetize your data.Get a Custom Demo or Ask our Experts
If you have questions or would like to request a customized demo to explore the use of Swarm Object Storage for your business or organization, contact us. We are ready to help!
As we head into the heart of summer, it reminds me of how quickly time flies and how rapidly technology changes. With a mature product such as Swarm, new features are added regularly to keep up with the evolving needs of the marketplace and our customers. In addition, we consistently review the value of existing features to ensure they not only meet current needs but anticipate future needs as well.
Two short years ago, my colleague Jamshid “Jam” Afshar blogged on how Elasticsearch & Object Storage solves petabyte-scale search as he prepared to discuss the topic in a webinar. Next week, Jam will be joining me on our monthly Tech Tuesday webinar to discuss using Elasticsearch with Object Storage.What is Elasticsearch and Why Should I Use it?
Elasticsearch is a distributed search and analytics engine that offers a RESTful API which can be used with object storage to enhance metadata searching operations.
In Swarm Object Storage, Elasticsearch provides the ability to list and query objects based on their metadata information. This is a key capability needed to bring structure to a large pool of unstructured data. (If you want to learn more about metadata with object storage, I highly recommend you watch our Using Metadata with Object Storage webinar.Why Does Swarm Object Storage Use Elasticsearch?
At Caringo, we were early adopters of Elasticsearch (going as far back as Elasticsearch version 0.90) because we needed a scalable solution to solve the problem of listing objects in a Swarm cluster. At the time, we evaluated NoSQL approaches including Solr, Elasticsearch and MongoDB in addition to traditional SQL database offerings (noting that traditional SQL databases lacked necessary scale and still do). We found that Elasticsearch was by far the most promising solution. Specifically, it passed our rigorous testing standards for speed of writes/updates and searches.
Additionally, Elasticsearch included an extensive API for management and diagnostics in an Elasticsearch cluster. Fulfilling the promise that we saw in the infancy of Elasticsearch, the technology has grown in popularity and reach with many large Elasticsearch deployments in production to date.How Does Elasticsearch Provide Structure to “Big Data?”
Swarm Object Storage software is fully integrated with Elasticsearch. This is implemented in the form of a “search feed” which populates the Elasticsearch cluster with the metadata information present on the stored objects. This information is effectively cached in the Elasticsearch cluster for fast list and query operations.
Furthermore, the Swarm API itself is extended to allow for list and query of Swarm objects in terms of their metadata. This results in the ability for Swarm to index object metadata in near real time, enabling you to perform ad hoc searches on the metadata attributes of stored objects. With the Swarm Content Portal, we take things further by providing a web UI which allows you to easily save frequently used queries as Collections. These Collections can be presented as virtual folders which will always return the latest set of objects that meet the criteria for the query.
Note that although Swarm Software is the ultimate authority for the metadata information of all objects stored, it’s still a best practice to take a snapshot of your Elasticsearch index. This allows for decreased time to recovery in the event of an unanticipated failure and allows you to quickly return list and query capability to clients and applications which depend on it.Ready to Learn More?
Register today for our July 16 webinar on Elasticsearch where Jam and I will:
- Explain what Elasticsearch is and the benefit of using it with object storage
- Take an in-depth look at best practices for using Elasticsearch with object storage
- Demonstrate the use of Elasticsearch with Caringo Swarm
The post Unleashing the Power of Object Metadata with Elasticsearch appeared first on Caringo.Related posts:
Independence Day (a.k.a, the Fourth of July) in the USA commemorates the Declaration of Independence (July 4, 1776), where the Continental Congress declared the American colonies as united, free, independent states. As we celebrate with fireworks, patriotic music and BBQ, the rest of the world goes on with business as usual. And, in today’s world, business means data—and lots of it!How Much Data is Being Stored?
What does this mean? You guessed it. The need for cost-effective, highly scalable storage with built-in data protection will only continue to grow.How is Data Stored?
Data is stored in a wide variety of solutions. From primary block- and file-based storage devices such as:
- SAN (Storage Area Network)
- NAS (Network-Attached Storage)
- DAS (Direct-Attached Storage)
To what is generally considered secondary storage:
- Cloud Storage (all of which is based on Object Storage technology)
- On-Prem Object-based Storage
- Tape Storage
If you want to learn more about the various types of data storage, I recommend you watch our Back to Basics webinar featuring CEO Tony Barbagallo and VP Marketing Adrian Herrera or read the blog What are the Differences Between Block, File and Object-Based Data Storage?Why Liberate Your Data?
Over the past years, we’ve talked a lot about data being locked in silos. It was one of the first blog topics I tackled when I started working for Caringo in 2015, inspired by Marc Staimer’s Ending Storage Silos whitepaper. The reasons for liberating data from silos and moving it into object storage remain the same:
- Improve your organization’s productivity with data portability between protocols (S3, SCSP, HTTP, HDFS and NFS)
- Expand search capabilities with metadata
- Lower the short- and long-term cost of storing data
- Support data storage and distribution at scale
- Increase resilience of data and simplify recovery
Register now for our August 20 Tech Tuesday webinar to learn about Data Resiliency & Recovery with Swarm Object Storage.
When your organization manages data effectively and can find data when it is needed, you have the freedom to focus on other aspects of business critical to your success. You can better collaborate, create, communicate and take care of your employees and customers.
When you free up staff hours by simplifying data management and dollars by reducing storage cost of acquisition and ownership (TCA and TCO), you create an environment ripe with possibility and innovation.
And, with Swarm Object Storage, you empower your Storage and IT Admin to unplug and find work-life balance.
Consolidating your data on our Swarm Object Storage Platform has never been easier. Check out our How to Migrate to Object Storage from SAN, NAS and Tape Storage resource page to learn just how simple we make it to liberate your data from silos and consolidate it on the Swarm Object Storage Platform for access, distribution and archive. If you need help, contact us. Our experts are happy to help.
How often do you go off the grid and really unplug? If you are a Storage or IT Admin, my guess is not often enough. The responsibility of maintaining a reliable and efficient storage environment for an organization is a heavy mantle. And, it becomes heavier as we increasingly rely on the data that is stored to run a business, create and/or deliver a product (for example, video content on streaming platforms).The Power of Unplugging
The power of unplugging from technology for people is well documented—in everything from lifestyle blogs to business articles and scientific research studies. We all know it is difficult in today’s competitive workplace; and let’s face it, some of us are workaholics who thrive by giving our careers 110%. But, we all know that we need to take a step back and go on vacation here and there, enjoy the holidays with our loved ones and indulge in a bit of “me” time.
However, the portability and convenience of mobile phones, tablets and laptops means we hardly ever leave them home or turn them off. These devices serve as a tether to so many important things in our life—family, friends, recreation and last, but certainly not least, our jobs. At Caringo, we may not necessarily all be good at unplugging, but we are all committed to making sure that our customers can unplug and not worry about the security of their data!4 Suggestions for Regaining Work-Life Balance for Storage and IT Admins
When it is literally your job to stay plugged in and you are on call around the clock, how do you unplug? While we cannot alleviate all the workplace concerns that might keep you up at night, we have a few suggestions that might help you rest easy about your data:
- Implement a storage environment with continuous, built-in data protection that is self-healing.
- Make sure your staff is properly trained.
- Have the right Support plan in place.
- Ensure that the health of your storage environment is monitored around the clock.
Object-based storage technology has some inherent benefits that make it valuable for many storage environments. As our VP of Sales Ben Canter mentioned in last week’s blog, Checklist for Evaluating Object Storage, that includes:
- Built-in data protection
- S3 compatibility
- Powerful metadata and search capabilities
With Caringo Swarm, we’ve enabled educational institutions such as Texas Tech University, been integrated into scientific research facilities such as the STFC Jasmine super-cluster in the UK, empowered the Media & Entertainment industry as they create and deliver a wide range of video and helped both private business and government organizations store everything from medical records to surveillance video.Who Can Help Me?
Remember those four suggestions to help you unplug above? Maybe they seem unattainable, but they aren’t. At Caringo, we can help you with each of those four items. Here’s how.1. Build your storage to have continuous data protection that is self-healing.
A best-of-breed object storage platform like Caringo Swarm will have continuous built-in data protection. With our market-hardened platform, you not only get that continuous data protection (detailed in the whitepaper Protecting Data with Caringo Swarm Object Storage), you get a storage cluster that is self-healing. The Swarm recovery process is automatic (other object storage products require a manual recovery process).
Hopefully you have a staff, but we know that for many small-to-medium businesses, one person shoulders the load for keeping IT functioning. That means 24x7x365, including vacations and holidays, you might be at least to some extent on call.
If you have a staff, providing them with the proper training is critical. That is why at Caringo we hold 3-day intensive Caringo Certified Training session for our customers to ensure they understand how best to use Swarm. Our training is conducted by our own engineering staff, all of whom have been involved in developing, installing and maintaining Swarm Object Storage. This enables our students to dive as deep as they want and to build lasting relationships with the most experienced object storage engineers in the industry. We share best practices gleaned from hundreds of object storage implementations, and we incorporate the feedback given to us in class into our technology and roadmap discussions.3. Have the right Support plan in place.
One of the complaints that analysts tell us they most often hear about technology products is that the Support is not adequate. At Caringo, we hear just the opposite. We make it our business to offer online self-serve knowledge resources and after hours emergency access to support for our product. We also offer Professional Services for when you need additional staffing resources or have complex storage changes that you want to undertake.4. Ensure that the health of your system is being monitored.
Whether you have internal or external resources monitoring the health of your storage system, it enables you to be proactive about detecting issues and adding capacity when needed. That is why we added Health Reporting to Swarm several years ago.Contact Us Today
If you need help getting your storage to the point where you feel like you can unplug and enjoy a vacation, contact us today. One of our object storage experts will be happy to discuss your use case to determine if Caringo Swarm Object Storage is the right choice for you. You can also visit our Getting Started page for more information.
The post How Caringo Swarm Object Storage Can Help You Unplug appeared first on Caringo.Related posts:
I love shopping for a new car. The new car smell is more addicting than just about anything else out there. But as my wife will attest, the process starts many months ahead of the actual purchase. I research extensively to make sure I am choosing the car that best fits my needs of today and the next few years. One key step in the process is developing the list of must haves (high safety ratings, space for a 6’ back seat passenger, heated seats) and want to have (navigation, heated steering wheel, good gas mileage). With my list in hand, I am able to eliminate options very quickly until I find the few that fit my needs the best.
In working with thousands of customers over the years, I have found when customers take a similar approach to storage solutions as I have with my car buying, they are guaranteed to leave happy. While we as vendors are experts on our solutions, only the customer can be experts on their “must haves” and “want to have.” For those evaluating object storage, I’ve put together a checklist along with some helpful tips and reference materials.
The most important question to start with is “what is the problem you are trying to solve?” Assuming the answer is that you need storage for your data, we recommend reviewing a few relevant articles to make your decision process smoother.Overview:
- Build or Buy?
- Storage & Data Management
- Data Management Interfaces
- Service Interfaces
- Unified Namespace
- Metadata Features
There are a lot of choices for storage today. It is likely that you have already determined that object storage is a good fit for your use case because of the many benefits found in best-of-breed object storage solutions such as Swarm:
- Built-in data protection
- S3 compatibility
- Powerful metadata and search capabilities
If you are not certain that your use case is a good fit for object storage, we recommend that you contact us to talk to a storage architect or check out some of our educational resources on the topic:
- Storage Switzerland eBook: NAS vs. Object—Which is Best for Your Data Center?
- Tech Tuesday Webinar: How Does Object Storage Fit Into Your Infrastructure?
- Tech Tuesday Webinar: What Your Storage Vendor Isn’t Telling You About S3
Evaluating Object Storage for Your Use Case
In 2006 when we launched our first product, it was a simple choice as the options were limited. You could use Caringo or you could go to EMC to purchase Centera. Both of these products evolved from the work of Caringo Co-Founder Paul Carpentier, recognized as the inventor of content addressable storage (CAS).
Through the years, more and more companies started to incorporate object-based storage into their offerings with different levels of success. In 2012, we identified 5 “must haves” for object storage. This included:
- Symmetric Architecture
- Data protection for any size file, any number of files and any capacity
- Instant access, platform neutrality, NO proprietary databases!
- Cloud storage enablement
- Entire stack provided by one company
Fast-forward to 2019, and now there are a lot of object storage vendors to choose from, and that makes the task far more challenging! You not only need to identify the “must haves,” you must determine if the product will work with your existing infrastructure, meet your immediate needs and then grow with your organization or business.
Last year, I talked with Senior Consultant John Bell about this very topic on the Tech Tuesday Webinar: Evaluating Object Storage Solutions. We discussed a number of points in depth. Based on that discussion, we have put together a checklist that can assist you in your evaluation.Checklist for Evaluating Object-Based Storage Solutions
There are significant differences between object storage platforms. Here are some of the features that you should look at as you narrow the field of products you will take the time to evaluate.
- Does the product have automated rapid recovery?
- How much of the storage hardware is utilized for actual content versus overhead?
- How is the metadata stored?
- What are the performance characteristics of the product, and what are the demands for your use case?
- What level of availability do you need for search, sharing or streaming?
- What is the minimum and maximum capacity?
You should determine if it makes more sense for you to build your system (using software-defined object storage) or buy your system (that is, have a turn-key solution where you buy an appliance with the software. There are pros and cons to each. Here are a few things to consider when making this decision:
- Appliance Approach
- “Turnkey” solution
- May not be as flexible as necessary to meet certain requirements
- Units of purchase and associated licensing may also be inflexible
- Software Defined
- Requires more work up front (e.g., hardware sizing and purchase, integration etc.)
- Highly flexible in meeting specific requirements
- Units of purchase and licensing are also very flexible (typically “on demand”)
Look at the storage and data management features and compare the products you are most interested in. Here is a list of features you will most likely want to investigate:
- Combination of UI and API
- “Single Pane of Glass” Web Management Portal
- Monitoring and Event Notification
- Automated Failover and Recovery
- Full Availability
- Capacity On Demand
- Volume Portability (This is a key feature not found in “object on file system” or similar solutions!)
How will you manage the data? I suggest you look for a system that offers you the following:
- “Browse and Query” (Content Portal and API)
- Flexible Protection Schemes
- Combination of Replication and Erasure Coding
- Protection Policy Range (global default to individual object)
- Usage Metering and Quota Support
- Identity Management Integration (Including support for multiple IDM stores)
- Access Control
- Management Delegation
What connections will you need between your Object Storage and your other storage devices or services? Make sure to investigate this thoroughly and outline your requirements clearly.
- Native API (RESTful API based on standard HTTP 1.1)
- S3 (Ideally, a superset of what is found in Amazon S3)
- NAS (NFS & SMB)
- Service Connectors (Public Cloud)
While this feature is often overlooked, it is quite important if you want your Object Storage to function efficiently. For a true Unified Namespace, the storage must have:
- Ability to reference the object stored by the same name…
- Independent of how it was created
- Regardless of how it’s being requested (S3, NFS/SMB, Native API etc.)
- Specifically, names that are “human readable”
- Can be done with UUIDs, but this isn’t user friendly
- Allows for alignment of naming conventions across multiple protocols
- Provides automatic synchronization of name changes
Make sure to look at how the storage system manages metadata, as it is key for keeping your data searchable and accessible.
- Should include standard metadata support for system management and basic object query
- Ideally includes comprehensive support for custom metadata
- “Unlimited” custom metadata
- Ability to list and query on custom metadata
- Collections (saved queries) are a powerful tool for dynamic data/object sets (Ability to surface Collections through multiple access protocols is highly desirable.)
- Metadata should be easily managed and protected by the storage itself!
- No separate metadata servers
- No specialized controller nodes
As you examine products, make sure that you will have the right level of support in place, along with the professional services and training you need.
- Commercial Support vs. “Do It Yourself”
- Portals for software access and knowledge base
- Outsourced monitoring and notification
- Professional Services
- Requirements gathering
- Training (on-site, online etc., including Certification)
If you have questions or want to discuss how to get started with object storage, contact us. My team and I are ready to help.
“Data is the new oil.”
—Clive Humby, Mathematician
Businesses lose billions of dollars a year to IT downtime, and the actual loss of digital files and data can be even more disastrous. Today’s business models—across all types of verticals and use cases—are all in some way powered by data. Take the breadth of data that we are collecting. Then, combine that with the multiple formats we store data in. Add to that equation the plethora of storage technologies that have been in use over the past century. Got the picture? Yeah, it’s a bit messy.How Do You Refine Data?
Much like crude oil, raw data isn’t necessarily useful. So, just how do you refine your data stores? Much like oil, data needs to go to a refinery. So, the first step is to contain data in a store that gives you the functionality you need. Depending on the type of data and the amount of data, there are many storage technology options you can choose from. However, if you have a large amount of data, particularly unstructured data, the best data refinery for you is likely going to be some type of S3-compatible object storage solution. I recommend you watch our Tech Tuesday webinar: What Your Storage Vendor Isn’t Telling You About S3 to hear what you should ask potential storage vendors.Why Pool Data in an Object Storage Solution?
One of the fundamental benefits of using an object storage technology is that you can pool massive amounts of data into one repository, thus eliminating the archaic “data silos” that many organizations still struggle with.
Once you have pooled that data, you open up all sorts of possibilities. You can start to extract value from your assets and business intelligence from your conglomeration of data.How Does Object Storage Work?
If object storage is the refinery, how does it store, protect and manage data? If you want to refresh your memory about how object storage is different from other types of storage (e.g., block and file), watch our Back-to-Basics Webinar or read our Back-to-Basics blog. You may also want to check out the Storage Switzerland eBook: NAS vs. Object—Which is Best for Your Data Center?Object Storage Protects Assets and Data
In last week’s blog, I mentioned the footage of Elton John that was used in the pre-show to the rock biopic Rocketman. This week, The New York Times reported that decades of Universal Music Group treasures burned in 2008 with casualties including original recordings from stars such as Ella Fitzgerald, Aretha Franklin, Elton and Nirvana. What could have been done to prevent this catastrophic loss?
Data protection is a core function of Swarm object storage. Swarm leverages cluster resources to protect data all the way from bit errors to natural disasters. While many storage vendors tell you that data loss in storage systems is completely avoidable, the truth is that it is right up there with death and taxes.
However, by selecting the appropriate storage system for your assets and data and applying appropriate parameters, you can minimize the probability of data loss. To understand how Caringo Swarm protects data, read these whitepapers:Metadata enables the refinement of data
Metadata, that is, data about the data, enables endless possibilities to unlock information and identify trends that can transform your business. In Caringo Swarm, our metadata is directly stored with the object. Learn more about metadata by watching our Tech Tuesday webinar or reading the summary.Elasticsearch lets you search metadata
Elasticsearch is a distributed, RESTful search and analytics engine that, when used with object storage, enhances metadata searching operations. Each Search Feed indexes metadata in Elasticsearch. In Swarm, Search capabilities map one to one with S3 metadata. This brings a number of benefits such as:
- The ability to derive actionable insight from targeted analysis
- Dynamic organization of content using classification, key words and descriptive content, with multiple ways to track that content
- Integrated search stack optimized within the storage system
When it comes to managing all of that data, you need the right tools for the job. Over the years, we’ve worked to make that task simpler and more efficient. To learn more, check out these three Tech Tuesday webinars:
- Using the Swarm Object Storage Content Portal UI
- Using the Swarm Object Storage Content Portal
- Monitoring Swarm Object Storage Using Prometheus Exporter & Grafana
With different types of businesses and organizations, the requirements for storage and the appropriate strategy vary. Carefully architecting your solution and doing a proof of concept to ensure compatibility with your existing systems is an important step as you investigate new technologies and solutions. If you have questions, I’d like to offer you the option to do what I do: talk to our experienced Object Storage experts. Just contact us and we will be happy to answer your questions or set up a customized demo for you.
Here in Austin, TX, the kids are out of school, temperatures are starting to soar and it is time to escape that heat by heading for the movie theatre. Rotten Tomatoes Rocketman Critics Consensus described the movie by saying, “It’s going to be a long, long time before a rock biopic manages to capture the highs and lows of an artist’s life like Rocketman.”Pre-Show Entertainment at The Alamo Drafthouse
At the Alamo Drafthouse (a little movie chain that started up in Austin over 20 years ago), the movie pre-show consists of many interesting clips from Elton John performances, and even William Shatner interpreting the lyrics to Rocketman for a 1978 SciFi Awards Show after being introduced by no other than Bernie Taupin, Elton’s longtime lyricist. (The link shows a clip on YouTube for those of you with five extra minutes to spare and are feeling particularly brave today.)No Spoiler Alerts Here
Once the movie begins to roll, you see a highly stylized and intense film that takes you on the rollercoaster ride of Elton John’s life, providing tremendous insight into his artistry and the contributions that he and Bernie Taupin have made to music, culture and community over the past fifty years. If you love the music of Elton John, run—don’t walk—to see this on the big screen.What Does Rocketman Have to Do with Object Storage?
I’m glad you asked. First, all that original footage used in the movie pre-show had to be recovered from some type of storage solution (likely, much of it was sitting on tape or in the cloud). Secondly, and I promised no spoiler alerts, the special effects in the movie are spectacular. Not in an action-packed, Marvel Universe or DC Comics style, but in an artistic and passionate way that makes this film gut-wrenching one moment and inspiring the next.
That, of course, leads me to ask the question, what technologies enable the sharing of historical events and the making of new films? How does Object-Based Storage like Caringo Swarm fit into the picture? (Pun intended.)How Can Object Storage Help?
At Caringo, we see the need for storing digital video growing daily, and we hear from post-production houses and visual effects editors that they need a cost-effective, reliable platform for archiving footage that is searchable and provides instant access to video clips. They want to be able to seamlessly tie this into asset manager integrations such as CatDV, Marquis Project Parking, Cantemo and Vidispine. And, just as importantly, they want to safely store these assets indefinitely.
Understanding Object Storage Technology
Whether retrieving historical footage or creating new films, enabling efficient file movement is critical, as is being able to find and retrieve clips. Learn more about how Caringo Swarm plugs into asset management solutions with the S3 API by reading this blog or by watching our recent webinar: How to Enable Video On-Demand in Workflows.
Check out our many educational object storage resources designed to help you understand Object Storage technology and the many benefits it can bring to your business or organization. Having pioneered Object Storage technology, we have a staff of highly experienced Object Storage Engineers who are happy to talk to you about your specific needs and help you architect the solution that will work for your environment. As a bonus, with our continuous built-in data protection and easy-to-manage data storage platform, Swarm Object Storage will leave you with the peace of mind and time to go soak in some of that summer fun!
At Caringo, we pride ourselves on making something complex—i.e., storing and accessing TB–PBs of data and billions of objects on heterogeneous hardware—
easy to manage.
The key to this is visibility into system status. We have had the option to use SNMP and Nagios for many years; however, we kept getting requests for an intuitive way to monitor Swarm object storage using more current monitoring and visualization platforms.
Integrate Elasticsearch into Swarm Object Storage
We took the first step in this direction in 2016 when we first integrated Elasticsearch into Swarm. Elasticsearch is an open-source RESTful search engine built upon Lucene. This provided us with a scalable way to index, view and search system and custom metadata.
Launch Prometheus Node Exporter for Swarm
The next step to make visual system status possible happened in April of 2019 when we launched the Prometheus node exporter for Swarm 10. Prometheus is a popular open-source monitoring solution that enabled us to export Swarm-specific metrics to Elasticsearch.
Using Grafana Open-Source Metrics Visualization Platform
The third and final step was using Grafana—an open-source metrics visualization platform. We were able to leverage some of the existing Grafana templates to quickly visualize Swarm metrics over a customizable period of time. Our technical staff has been using this internally for a few months to optimize Swarm cluster configuration with excellent results.
Demo: Monitoring Swarm Object Storage with Prometheus & Grafana
In our June 11 TechTuesday webinar (at 7am PT/10am ET), Monitoring Swarm Object Storage Using Prometheus Exporter & Grafana, John Bell, Senior Consultant, will host Abraham “Avi” Felsenstein, System Integrator. Avi will demonstrate how to import data from the Swarm Prometheus node exporter and view it via Grafana.
The post Visualizing Swarm Object Storage Status with Prometheus & Grafana appeared first on Caringo.Related posts: