Caringo Blog

Subscribe to Caringo Blog feed Caringo Blog
One Storage Platform for Access, Distribution & Archive
Updated: 36 min 50 sec ago

Object Storage at IBC2019

Thu, 09/05/2019 - 01:30

Next week, I am once again heading back to the IBC show in Amsterdam, considered by some to be the world’s most influential media, entertainment and technology show. For me, a large part of the IBC experience is created by the unique venue of the event (the RAI), the exceptional quality and brilliant diversity of the delegates and the outstanding city of Amsterdam, filled with splendid restaurants and bars.

About IBC2019

Slated for 13–17 September 2019, this year’s IBC has 400 inspirational speakers and boasts a world-class exhibition, packed with 1,700 exhibitors (including our Caringo team at Stand 5.C33). This provides a perfect environment in which to network, build relationships and discover the latest trends and technologies. I always find the sense of opportunity overwhelming, and this year, I am excited to discuss with attendees the latest developments at Caringo.

Expanding Caringo’s Global Footprint

Since last year’s IBC, Caringo has greatly expanded its global footprint with landmark deployments in Europe, the US and East Asia. In the UK (where I reside), Swarm emerged as the leader after object storage performance benchmarking at the Science and Technology Facilities Council’s (STFC) JASMIN super data cluster. (Download the STFC Object Storage Benchmarking Case Study & Whitepaper to learn more.)

Addressing More M&E Use Cases

As our pure object storage platform has been adopted by organizations and used for new use cases, the needs and input of our customers has helped to shape the evolution of our Swarm pure object storage platform. With our expansion of media & entertainment (M&E) use cases and customers, we have sharpened our capabilities to integrate into digital video workflows.

Over the past year, Caringo Swarm has been adopted by elite streaming services and implemented as an on-demand archive for professional sports teams.

Visit with Caringo at IBC2019

Join us at Stand 5.C33 in the IBC exposition to chat with me and our other storage experts. We will share with you how our upcoming Swarm 11 release will revolutionise your workflows and help you collaborate and scale your content with speed and simplicity. If you would like to obtain our code for a complimentary IBC expo pass or to make an appointment with us at the event, contact us.

Learn More About Caringo Swarm Object Storage

Can’t make it to IBC this year? Join VP Marketing Adrian “AJ” Herrera and Director of Product Eric Dey for a look at Accelerating On-Demand Access to Video Archives with Swarm 11. They will show you what’s new in Swarm 11 with a live demonstration of our newest features and interactive Q&A.

Register now

The post Object Storage at IBC2019 appeared first on Caringo.

Related posts:
  1. It’s a Bird! It’s a Plane! It’s Swarm Object Storage!
  2. Papers & eBooks
  3. What are the 5 Tiers of Storage for New Video Production Workflows?

How Object Storage Optimizes Your Time & Resources

Thu, 08/29/2019 - 01:30

How valuable is your time?

If you are in the M&E industry (or deal with digital video), what is the value of being able to complete more projects, send videos to employees/editors faster or stream more videos to subscribers?

Once organizations grow to the level where their workflows are being strained, they often look to remedy that starting with the front-end application. Once they get to the storage layer, they are usually looking for the most economical storage to satisfy the requirements of the front-end application. This often boils down examination of storage to a financial ($/GB) rather than a functional analysis. The problem with this approach is that the functional value added by object-based storage platforms like Caringo Swarm gets lost.

For object storage specifically, the time and resource optimization value gets lost. I often hear object storage positioned as cheap and deep storage, which I think is a misnomer.

How Does Swarm Object Storage Go Beyond Cheap and Deep?

Caringo Swarm object-based data storage is scalable and strategic for asset access. That is, it provides extensive benefit strategically by serving up effortless scale for rapidly growing archives of data and granting instant access to those assets.

Wait, What About Tape Storage?

While tape will always be cheaper and deeper and is perfectly adequate for files and assets you may not ever retrieve, it lacks one of the most beneficial features that you get with an object storage platform like Swarm: on-demand access.

Scalable Object Storage

We have a lot of excellent resources that go into detail on how scalable object storage is. Two that I recommend are:

Now, let’s go into how object storage has evolved as a strategic tool for asset access.

Archive, Active Archive and On-Demand Archive

When you look up the definition of archive, you will most likely find it associated with the word “preservation.” Historically, archive has been a backup or a copy of an asset made for when and if that asset is needed again. In the storage world, it has long been clear that there is a large portion of digital assets that need to be continually accessed, leading to the term “active archive.” The word active denotes access but doesn’t specify time to access.

After talking to many in the M&E and Digital Video Industry, it is clear that time is their primary issue. Specifically, the time to move video from remote locations into their network, the time it takes to move working files to different locations, the time it takes to restore files for reuse from archives and the time it takes to efficiently stream content to internal or external viewers. This is why at Caringo you will often hear us use the term “on-demand archive.” On-demand denotes immediate access. You don’t need to wait for a video to be found, a tape to load, or wait for the mail to arrive with the thumb drive or hard drive. Your archived video is always instantly accessible to your applications, internal employees and (if you authorize access) to the public for on-demand viewing.

How Does Swarm Object Storage Save You Time?

In addition to providing immediate access to archived video, Swarm object storage also enables multi-tenancy, content and metadata management with integrated search. All of these features save both your administrators and end-users time by providing an intuitive web-based platform where they can have direct access to their portion of storage (private cloud storage). They are able to search the assets they are authorized to view and can also update assets by adding relevant metadata.

How do You Measure the Value of Time in Your Organization?

Where the rubber meets the road is how you measure or assign a value to saving time. What if you could complete 10–20% more projects per month? What if you could spend 50–60% less time managing access to content across your organization or with clients? How about if you could provide perpetual access to a specific portion of your archived video for a new VOD (video-on-demand) service? However you place value on time, I can tell you this…it’s not by looking at a $/GB calculation!

Swarm 11, Focused on Optimizing Time

If you are interested in learning more about how Swarm object storage optimizes your time, look no further then our next webinar. Eric Dey, Director of Product Management, and I will discuss Accelerating On-demand Access to Video Archives with Swarm 11.

And as always, we have a team of experts at Caringo that can help you figure out how to measure the value of plugging in an on-demand archive like Swarm into your workflows. Contact us if you would like to setup an overview.

The post How Object Storage Optimizes Your Time & Resources appeared first on Caringo.

Related posts:
  1. What are the 5 Tiers of Storage for New Video Production Workflows?
  2. S3 Eval
  3. Object Storage: Enabling On-Demand Workflows for Sports Video

It’s a Bird! It’s a Plane! It’s Swarm Object Storage!

Thu, 08/22/2019 - 01:30

I travel a lot for work. July alone, I was on a plane every week. New York, Austin, DC and Chicago were a few of the stops I made to visit great customers and partners like CatDV, Masstech, Diversified, AWS and Cinesys. And with each negotiation through an airport, I witnessed the same struggle with each passenger…

control of their content.

The Balance Between Content and Storage Space

For some travelers, it was an epic battle to see if they could get their bag into the ever-shrinking overhead compartments on airplanes. Overwhelmed airline attendants do their best to help, but they have to juggle the enforcement of the rules (size of bags, # of bags, etc.) and customer happiness. Add in the fact that some customers get preferential treatment because of status (first class, frequent traveler, cost paid for a seat, and so on).

Watching this scenario play out reminded me of the struggles many organizations face every day with storing data. I know, I know…how do flight attendants and passengers and luggage battles remind me of data storage, and specifically, Caringo’s award-winning Swarm Object-Based Data Storage?

Please allow me to explain. But I have to warn you: grab your tissues now because there could be tears.

What is Happening in the Content World Today?

Files are getting larger, people are deleting less and access is happening over more devices than ever before—these are three things I think we can all agree are happening in the content world today.

As resolution increases the resulting files are jumping in size, more cameras are being added to the world, and mobile devices are becoming more powerful (and 5G supported), making the dynamics of storing and managing content harder than ever.

How Can Swarm Object Storage Help with Storing Data?

Swarm provides a platform that allows users to store all content and to easily search for exact items using traditional means (file name, date, location) or more modern ways (that is, metadata search). Swarm also allows you to retrieve the exact data you need. This is similar to the well-organizer traveler who has the items they need most in the seat pocket in front of them.

If you think of the flight attendant as the administrator of the plane—making sure only the correct passengers enter the airplane and ensuring each passenger only uses their allotted space but in a self-service manner. That enables passengers use the system in an organic way that fits their travel flow.

Swarm does all these same things. Simply put, Swarm creates an easy-to-manage storage platform that allows content to be efficiently accessed. With powerful built-in metadata search, it’s easy to find whatever it is that you need in the object store. You can restore a whole clip or just a few frames as you need to. Best of all, Caringo Swarm slides content right into your existing workflow. And, given that it is less expensive than public cloud and more reliable than tape, Swarm Object Storage will be the smartest investment in your environment.

Buckle Up for IBC2019!

This September, I’ll be heading to IBC2019 at the RAI Amsterdam 13 – 17 September. IBC (originally the International Broadcasting Convention) has evolved from its technical broadcast roots to encompass the entire breadth of media creation management and delivery. Visit us at IBC Stand 5.C33 to speak with our object storage experts to hear what’s new in Swarm 11 or register now for our upcoming webinar: Accelerating On-Demand Access to Video Archives with Swarm 11.

The post It’s a Bird! It’s a Plane! It’s Swarm Object Storage! appeared first on Caringo.

Related posts:
  1. Object Storage Delivers Archive Access and Agility for M&E Workflows
  2. Papers & eBooks
  3. FAQs

Selecting the Right Data Storage Tools for the Job

Thu, 08/15/2019 - 01:30

“The best investment is in the tools of one’s own trade.”

Benjamin Franklin

In today’s fast-paced world, the data storage landscape changes at a blindingly fast pace. It seems that every day there is a new start-up claiming they have “invented a revolutionary new way to store and manage your data.” But, when it comes down to it, many are just re-inventions building on an existing archetype (that is, someone else’s ideas with a bit of a “twist”).

What are the Different Types of Storage?

There are three primary ways to store data. File, Block and Object storage. For more information on this topic, read our Back-to-Basics blog.

Once you understand the different types of storage, it may become obvious which type of storage you need to add to your existing infrastructure or deploy for your use case.

How Does Object Storage Fit Into Your IT Infrastructure?

Often, object storage is used to augment your existing storage, whether that be SAN or NAS. In our on-demand webinar How Does Object Storage Fit into Your IT Infrastructure?, our CEO Tony Barbagallo and Sr. Consultant John Bell take on this topic. They review common use cases for object storage and talk about how it fits into your existing storage network.

Should I use Object Storage Instead of SAN, NAS or Tape Storage?

Object storage is ideal when content will be accessed again or reused, especially over the web (e.g., digital video or digital evidence). Object storage is also useful when you need a cold archive but want the data to be more secure and simpler than to retrieve than tape.

However, if you are running a highly transactional workload, you will want to use SAN or NAS as your primary (or tier 1) storage.

The type of storage you need varies by use cases. Data storage is never a one type and size fits all proposition (and beware if a vendor tells you that their data storage is a good fit for every scenario).

In the chart below, we show which type of storage suits various use cases.

5 Signs You Need Object Storage

In my blog NAS vs. Object 2019: If Not Now, When?, I outlined the telltale signs that you need to add Object Storage into your existing infrastructure. Here is a high-level recap:

  1. You have lots of unstructured data to store that may be accessed again
  2. Your primary storage devices are overloaded and slow
  3. It is difficult to locate data
  4. You want to intelligently mine your data for insights
  5. Your IT budget is tight and your storage needs are high
Twists in Object Storage: What Makes Swarm Object Storage Different?

In the data storage industry, tools can be quite nuanced. So, understanding the twists (aka, the features and functionality) of various products is critical in making your product selection. Here’s just a few of the things that set Caringo Swarm apart:

Stability: Having launched our first product in 2006, our Swarm Object Storage is field-hardened and highly reliable.

Scalability: Scaling from small to 100s of Petabytes is one of the hallmarks of our pure object storage approach as well as a testament to our long-standing relationships with some of our earliest customers.

Metadata: By storing metadata with the object rather than in a separate database (a method employed by most object storage vendors), we unleash the full potential of metadata. To learn more, hear what Ryan Meek, our Chief Solutions Architect has to say on the topic in our Using Metadata with Object Storage webinar.

Support: Our Support team has some of the most experienced storage engineers in the business and with a global presence, help is never far away.

Resilience & Recovery: Undoubtedly, this is one of the most important factors in choosing a data storage solution. Register now to attend our August 20 Tech Tuesday webinar live as VP of Engineering T.W. Cook joins Sr. Consultant John Bell to review just how Swarm keeps your data safe and recovers it when needed. They will be taking questions throughout the broadcast.

(Note: All of our webinars are recorded and available on demand after the broadcast concludes.)

How Can Caringo Help?

“There is a great satisfaction in building good tools for other people to use.”

Freeman Dyson

We love what we do because we see the benefit that our Caringo Swarm Object Storage and accompanying products such as SwarmNFS, FileFly, Caringo Drive and Single Server bring to organizations. We know that not every organization has a storage architect on staff, so at Caringo, we have a number of highly qualified storage architects, consultants and engineers that are happy to help you. Just visit our contact page.

The post Selecting the Right Data Storage Tools for the Job appeared first on Caringo.

Related posts:
  1. What are the 5 Tiers of Storage for New Video Production Workflows?
  2. Checklist for Evaluating Object Storage
  3. How to Migrate to Object Storage from SAN, NAS and Tape Storage

Object Storage Delivers Archive Access and Agility for M&E Workflows

Thu, 08/08/2019 - 01:30

I was recently at the SVG (Sports Video Group) Content Management Summit in New York City where our CEO Tony Barbagallo was on a panel titled Object Storage for M&E: Making it Work for Broadcasters On-Prem and in the Cloud. Right before the panel started, one of the attendees asked the person sitting next to him, “What is object storage?” The first person who answered worked for a large file transfer company. He defined object storage as a type of storage target that a data manager can write to. Once he finished, I interjected with a brief description of the technical differences between file, block and object storage.

What is Object Storage?

After seeing the expression on the face of the attendee who originally asked the question, I realized that I had also missed the mark. So, I launched into explaining object storage using the popular analogy of a valet parking system. In this analogy, the content is your car and all you need is the ticket to retrieve that content. While that explanation resonated with him, there was still clearly a disconnect between the technical aspect of easily retrieving a file and the benefits of using object storage.

Perspective is Everything

Three things became obvious:

  1. Whoever is defining a technology will do so from their own perspective.
  2. The person looking for the definition will process the definition based on their own experiences.
  3. We need a way to describe the actual benefit of object storage to the end user.

Armed with these revelations, it became clear that I needed to come up with a new analogy for object storage that explained the benefits of the technology from the end-users’ perspective (e.g., a subscriber of a video streaming service or a viewer of sports video), rather than from the perspective of a Storage Infrastructure Manager or IT Executive.

Pure Object Storage Delivers Access and Agility

In my quest to come up with a new analogy to illustrate the benefits of using object storage, I thought about the process of packing for a trip. For people who travel extensively, luggage can become an obsession and packing an art. After all, the success of your trip depends on having the right items with you, just as the success of your business endeavor relies on being able to access or stream content rapidly.

There are a number of tricks to packing quickly and efficiently as you need to grab the correct items for each trip. After all, what is a beach getaway without casual clothes, swimsuit, sunscreen and comfortable footwear? Or a business trip without proper attire and grooming products?

Think of your suitcase as the video you plan to view, whether it is from a PC, mobile device or connected to a device such as Roku, Fire TV Stick or Apple TV). The combination of the items in your suitcase are like the bits of data needed to display a video file. With object storage, your suitcase is always packed and flight ready, so your video is instantly available for delivery over the web.

How Does Object Storage Compare to File Storage?

Whereas object storage resembles a pre-packed, flight-ready suitcase, file storage requires that a suitcase be packed every time that video is requested.

Imagine that the items you need to pack in your suitcase are strewn around your house and you must go fetch each and every one prior to packing. This is similar to how file storage works. With file storage, the bits and pieces you need for the video are spread across a storage device, and they must first be assembled. Then, the assembled file needs to be sent to a web server to deliver over the web.

Object Storage Accelerating Archive Access and Agility

Streamlined video access is just one of the many benefits of pure object storage that we have been working on for over a decade. And we are not done. On September 10 we will be launching Swarm 11, which is focused on accelerating archive access and agility. Our partners and customers can request a preview before launch. To do so, just reach out to your Caringo Representative. If you are not an existing customer or partner, make sure to sign up for our monthly Caringo Buzz newsletter and we will send you an invitation to one of our upcoming launch webinars.

The post Object Storage Delivers Archive Access and Agility for M&E Workflows appeared first on Caringo.

Related posts:
  1. Object Storage: Enabling On-Demand Workflows for Sports Video
  2. Dodgeball, Teamwork and the 5 S’s of Storage
  3. FAQs

Resilience & Recovery with Swarm Object Storage, part 2

Thu, 08/01/2019 - 01:30

Data recovery in Swarm works hand-in-hand with its data resiliency features. In our last blog post, we discussed how Swarm distributes and protects data from the possibility of failure. When a failure actually happens, Swarm performs active recovery with the goal of again achieving full protection for its data. We regularly perform a kind of “whack-a-mole” test with up to 14 sequential failures. A properly configured cluster can have multiple simultaneous hardware failures and achieve full replication again in minutes.

What Happens When a Storage Volume Fails?

Of all the types of failures, volume failures are perhaps the most common. Even with annualized volume failure rates of a low 2%, a large Swarm cluster (e.g., PB+) can potentially have one or more failures a month. Because of Swarm’s parallel active recovery mechanisms, a larger cluster will also recover from such a failure faster than will a smaller cluster. Few storage solutions in the market scale its recovery mechanisms in this way. Swarm’s active recovery means data loss only happens with true catastrophic failures or gross neglect.

How Does Swarm Recover?

Volumes can be physically moved from one chassis to another, either newly introduced or already running in the cluster. The data on the volume will then be available for use after mount. While Swarm can recover from the permanent loss of a chassis (along with its volumes), chassis loss is usually a temporary problem remedied by a new power supply or network card.

Performing full recovery in a chassis or subcluster loss may be counterproductive if the data is known to be on viable disks. In such a situation, Swarm can alert to this condition and accept administrator guidance on whether full recovery of all volumes should be performed. In the absence of this feedback, Swarm will proceed with the recovery. As described in part 1 of this blog, Swarm will be able to read and write all cluster data in either of these temporary loss scenarios.

Best Practices for Cluster Configuration

Some Caringo customers keep multiple clusters for high availability and failover. Clusters can be easily configured for remote replication and multiple clusters can be configured to mirror each other. With small network configuration changes, either cluster can be the primary access for an application or user base or load balancing to both clusters can be used to serve out exactly the same objects. Should one of those clusters suffer a catastrophic failure, the other cluster can carry the load while the first one is repopulated.

Take the Worry Out of Protecting Data

With its combined data resilience and recovery mechanisms, Swarm takes the worry out of protecting your data and making it always available. These mechanisms “just work” with little configuration or manual intervention. This is especially important with remote data centers that may only be periodically serviced.

Learn More About Data Resilience & Recovery

Register now for our August 20 Tech Tuesday webinar, Data Resilience & Recovery in Swarm Object Storage. It will feature T.W. Cook, VP Engineering, and John Bell, Sr. Consultant, and include live Q&A throughout the webcast.


The post Resilience & Recovery with Swarm Object Storage, part 2 appeared first on Caringo.

Related posts:
  1. Data Protection & Active Data Recovery
  2. Resilience & Recovery with Swarm Object Storage, part 1
  3. Protecting Data in the Cloud Age with Object Storage, part 2

Resilience & Recovery with Swarm Object Storage, part 1

Thu, 07/25/2019 - 01:30

Data resiliency has been a feature of Swarm from the beginning. The architecture of Swarm object storage has never had a single point of failure. One or more Swarm nodes may be down for maintenance or due to hardware failures and the rest of the Swarm cluster keeps running, servicing requests.

Since the data in Swarm remains available, users are likely to not even know about the outage. Because a larger Swarm cluster has better throughput than a smaller one, the loss of one or more nodes generally only has a small impact on performance.

Why is Swarm Object Storage Resilient?

One of the resiliency features is that Swarm keeps multiple replicas of objects and it will not co-locate replicas where single failures are likely to take them both out. It’s the old idea of “don’t put all your eggs in one basket.” Replica distribution is discussed in my 2016 blog series (Protecting Data in the Cloud Age with Object Storage) as well as in our Protecting Data with Caringo Swarm Object Storage whitepaper.

The level of protection is easily configured, both at the cluster and at the object level. As a result, Swarm and the data it stores are resilient to failures at platter level, the level of the disk drive, the chassis, and even logical subclusters. In the latter case, customers can configure Swarm to know about a cluster’s network topology and location information.

How Replication and Distribution Increase Resiliency

Full replication and distribution of replicas (and erasure coded segments) are achieved from the time of the initial object write. Swarm’s health processor is a background task that is continuously checking for the presence and locations of replicas so that a cluster’s data is always protected from all but the most catastrophic failures, such as a fire or flood. Various remote replication options provide data resiliency for those possibilities. You can learn more by reading the Elastic Content Protection Technical Overview paper.

Learn More about Resilience & Recovery of Swarm Object Storage

Data resiliency is an integral part of Swarm object storage software and we are proud to have customers with clusters that have been operational for over 10 years, through all manner of hardware failures and upgrades. In next week’s blog post, I’ll discuss the data recovery features that come into play when a failure actually happens.

Register now for our August 20 Tech Tuesday webinar, Data Resilience & Recovery in Swarm Object Storage. It will feature T.W. Cook, VP Engineering, and John Bell, Sr. Consultant, and include live Q&A throughout the webcast.


The post Resilience & Recovery with Swarm Object Storage, part 1 appeared first on Caringo.

Related posts:
  1. Protecting Data in the Cloud Age with Object Storage, part 2
  2. Data Protection & Active Data Recovery
  3. Protecting Data in the Cloud Age with Object Storage

Illuminate Your Dark Data with Object Storage

Thu, 07/18/2019 - 01:30

Dark data. Sounds a bit sinister, doesn’t it? Let’s unravel the mystery of dark data and talk about the complex issues that must be considered and the ramifications of storing or not storing that data for businesses and organizations.

What is Dark Data and Should You Bother to Store It?

Wikipedia defines dark data as data “acquired through various computer network operations but not used in any manner to derive insights or for decision making.”

The volume and rate of collecting data can easily exceed the capability of most organizations to properly tag, store and analyse that data. Not surprisingly, given how difficult it can be to identify the wheat from the chaff, it has become a common practice to store all the data that is generated.

With over 2.5 quintillion bytes of data created every single day, and an estimated 1.7MB of data created every second for every person on earth by the next year, this becomes an increasingly pressing issue for storage architects and IT departments.

Storing all that data creates everything from compliance issues to overburdened storage systems and also raises the possibility of ransomware threats, but deleting it has the potential to cause even more problems. What if you accidentally delete something you need or that might prove to be invaluable later on?

So, most organizations continue to add more storage as they accrue data, and much of that data is unstructured data.

What is Unstructured Data?

You could say that enabling organizations to cost-effectively store unstructured data is our business. After all, we’ve been doing that since our first product release in 2006.

Unstructured data is quite abundant in today’s IT landscape. It can be just about anything, from music recordings to medical imaging to video footage. The defining characteristic of unstructured data is that it is not stored in a structured, predefined format. That makes it challenging not just to store, but also to manage.

How do you Store and Manage Unstructured Data?

Over the years, the Caringo team has helped numerous customers store, organize and access massive amounts of unstructured data with our Swarm Object Storage Software. Check out our case studies for detail about how we helped organizations like the STFC Scientific Computing Department, Texas Tech University and NEP in the Netherlands.When you tier data into Swarm object-based data storage, you benefit from continuous built-in data protection, management, organization and search at massive scale. As the pioneer in object storage technology, Caringo products have some distinct differences in methodology that give our customers a significant advantage. While we cannot cover them all in just one blog, part of the Swarm difference revolves around our integration of Elasticsearch and how we store and use metadata (a.k.a., data about the data).

To learn more, watch our Tech Tuesday webinar about using metadata with object storage on demand or read the summary that follows the webinar.

How Does Caringo Use Metadata & Elasticsearch to Illuminate Dark Data?

Metadata and Elasticsearch are the key to making data easy to find in Swarm Object Storage. This is a topic we have addressed in a number of blogs and webinars, including our most recent Tech Tuesday webinar (using elasticsearch with object storage).

Using Swarm’s extensive custom metadata capabilities and Elasticsearch simplifies the task of locating discrete types of data in a large data store. It gives you dynamic organization of content with classification, key words, descriptive content and multiple methods to track content with no separate big data project required.

Once you have data in a Swarm Object Storage cluster, the content of it and the value of it are illuminated, so you can reap insights and potentially realize new ways to monetize your data.

Get a Custom Demo or Ask our Experts

If you have questions or would like to request a customized demo to explore the use of Swarm Object Storage for your business or organization, contact us. We are ready to help!


The post Illuminate Your Dark Data with Object Storage appeared first on Caringo.

Related posts:
  1. How Do I Use Metadata with Object-based Data Storage?
  2. Papers & eBooks
  3. Unleashing the Power of Object Metadata with Elasticsearch

Unleashing the Power of Object Metadata with Elasticsearch

Thu, 07/11/2019 - 01:30

As we head into the heart of summer, it reminds me of how quickly time flies and how rapidly technology changes. With a mature product such as Swarm, new features are added regularly to keep up with the evolving needs of the marketplace and our customers. In addition, we consistently review the value of existing features to ensure they not only meet current needs but anticipate future needs as well.

Two short years ago, my colleague Jamshid “Jam” Afshar blogged on how Elasticsearch & Object Storage solves petabyte-scale search as he prepared to discuss the topic in a webinar. Next week, Jam will be joining me on our monthly Tech Tuesday webinar to discuss using Elasticsearch with Object Storage.

What is Elasticsearch and Why Should I Use it?

Elasticsearch is a distributed search and analytics engine that offers a RESTful API which can be used with object storage to enhance metadata searching operations.

In Swarm Object Storage, Elasticsearch provides the ability to list and query objects based on their metadata information. This is a key capability needed to bring structure to a large pool of unstructured data. (If you want to learn more about metadata with object storage, I highly recommend you watch our Using Metadata with Object Storage webinar.

Why Does Swarm Object Storage Use Elasticsearch?

At Caringo, we were early adopters of Elasticsearch (going as far back as Elasticsearch version 0.90) because we needed a scalable solution to solve the problem of listing objects in a Swarm cluster. At the time, we evaluated NoSQL approaches including Solr, Elasticsearch and MongoDB in addition to traditional SQL database offerings (noting that traditional SQL databases lacked necessary scale and still do). We found that Elasticsearch was by far the most promising solution. Specifically, it passed our rigorous testing standards for speed of writes/updates and searches.

Additionally, Elasticsearch included an extensive API for management and diagnostics in an Elasticsearch cluster. Fulfilling the promise that we saw in the infancy of Elasticsearch, the technology has grown in popularity and reach with many large Elasticsearch deployments in production to date.

How Does Elasticsearch Provide Structure to “Big Data?”

Swarm Object Storage software is fully integrated with Elasticsearch. This is implemented in the form of a “search feed” which populates the Elasticsearch cluster with the metadata information present on the stored objects. This information is effectively cached in the Elasticsearch cluster for fast list and query operations.

Furthermore, the Swarm API itself is extended to allow for list and query of Swarm objects in terms of their metadata. This results in the ability for Swarm to index object metadata in near real time, enabling you to perform ad hoc searches on the metadata attributes of stored objects. With the Swarm Content Portal, we take things further by providing a web UI which allows you to easily save frequently used queries as Collections. These Collections can be presented as virtual folders which will always return the latest set of objects that meet the criteria for the query.

Note that although Swarm Software is the ultimate authority for the metadata information of all objects stored, it’s still a best practice to take a snapshot of your Elasticsearch index. This allows for decreased time to recovery in the event of an unanticipated failure and allows you to quickly return list and query capability to clients and applications which depend on it.

Ready to Learn More?

Register today for our July 16 webinar on Elasticsearch where Jam and I will:

  • Explain what Elasticsearch is and the benefit of using it with object storage
  • Take an in-depth look at best practices for using Elasticsearch with object storage
  • Demonstrate the use of Elasticsearch with Caringo Swarm



The post Unleashing the Power of Object Metadata with Elasticsearch appeared first on Caringo.

Related posts:
  1. How Do I Use Metadata with Object-based Data Storage?
  2. FAQs
  3. Elasticsearch & Object Storage: PB-Scale Search Solved

Liberate Your Data with Object Storage

Thu, 07/04/2019 - 01:30

Independence Day (a.k.a, the Fourth of July) in the USA commemorates the Declaration of Independence (July 4, 1776), where the Continental Congress declared the American colonies as united, free, independent states. As we celebrate with fireworks, patriotic music and BBQ, the rest of the world goes on with business as usual. And, in today’s world, business means data—and lots of it!

How Much Data is Being Stored?

Humanity generates a staggering 2.5 exabytes of data every day


the entire digital universe is expected to reach 44 zettabytes by 2020.

What does this mean? You guessed it. The need for cost-effective, highly scalable storage with built-in data protection will only continue to grow.

How is Data Stored?

Data is stored in a wide variety of solutions. From primary block- and file-based storage devices such as:

  • SAN (Storage Area Network)
  • NAS (Network-Attached Storage)
  • DAS (Direct-Attached Storage)

To what is generally considered secondary storage:

  • Cloud Storage (all of which is based on Object Storage technology)
  • On-Prem Object-based Storage
  • Tape Storage

If you want to learn more about the various types of data storage, I recommend you watch our Back to Basics webinar featuring CEO Tony Barbagallo and VP Marketing Adrian Herrera or read the blog What are the Differences Between Block, File and Object-Based Data Storage?

Why Liberate Your Data?

Over the past years, we’ve talked a lot about data being locked in silos. It was one of the first blog topics I tackled when I started working for Caringo in 2015, inspired by Marc Staimer’s Ending Storage Silos whitepaper. The reasons for liberating data from silos and moving it into object storage remain the same:

  • Improve your organization’s productivity with data portability between protocols (S3, SCSP, HTTP, HDFS and NFS)
  • Expand search capabilities with metadata
  • Lower the short- and long-term cost of storing data
  • Support data storage and distribution at scale
  • Increase resilience of data and simplify recovery

Register now for our August 20 Tech Tuesday webinar to learn about Data Resiliency & Recovery with Swarm Object Storage.


How Does Swarm Object Storage Provide Freedom?

When your organization manages data effectively and can find data when it is needed, you have the freedom to focus on other aspects of business critical to your success. You can better collaborate, create, communicate and take care of your employees and customers.

When you free up staff hours by simplifying data management and dollars by reducing storage cost of acquisition and ownership (TCA and TCO), you create an environment ripe with possibility and innovation.

And, with Swarm Object Storage, you empower your Storage and IT Admin to unplug and find work-life balance.


How do I Consolidate Data on Object Storage?

Consolidating your data on our Swarm Object Storage Platform has never been easier. Check out our How to Migrate to Object Storage from SAN, NAS and Tape Storage resource page to learn just how simple we make it to liberate your data from silos and consolidate it on the Swarm Object Storage Platform for access, distribution and archive. If you need help, contact us. Our experts are happy to help.

The post Liberate Your Data with Object Storage appeared first on Caringo.

Related posts:
  1. FAQs
  2. How Do I Use Metadata with Object-based Data Storage?
  3. Papers & eBooks