The value of defining your own search dataset

There are many web monitoring and listening platforms, both free and commercial options, available online. The majority of these companies advertise their ability to monitor and scour hundreds of thousands of sources simultaneously. These turnkey solutions allow the user to submit lists of keywords and the platform returns thousands of hits from blogs, message boards, social media websites, news websites, etc.  This type of solution works great for projects that simply need to track mentions of specific keywords, particularly keywords and sources that are used globally. However, this type of turnkey model using a standard database doesn’t work for everybody.

Many users need to define their own dataset rather than blindly trusting one provided to them.  In this post, we give examples of who should be defining their own data set, the value of monitoring your own data, and available custom monitoring solutions.

Who Should Define Their Own Data Set and What Value Does it Bring?

Data Image Thumbnail

  • Marketing professionals monitoring a brand in a specific region

Marketing professionals wanting to track a brand in a specific region must often sift through thousands of irrelevant false hits from all across the globe. Instead, select the specific state and local newspapers and blogs you want to monitor and select custom geographic areas on Twitter to collect tweets from.

  • Investment managers monitoring specific companies

Investment managers know certain sources of information are especially relevant when trying to monitor and track specific companies. Take advantage of defining your own sources and datasets by picking sources such as press releases, government regulatory sites (USPTO, FDA, SEC, etc.), news, and investor relations portions of sites.

  • Health researchers

Health researchers want to stay on top of what is happening in their field without having to read through every article in their favorite journals and health research news websites. Harvest from journals and news sources you select, and create keywords to alert you to only content relevant to you.

  • Any data analyst

BrightPlanet offers near real-time alerting of new and modified content from any web source for those that can identify where and what you are looking for.

Value of defining of your own source with BrightPlanet

  • Get More Data

BrightPlanet’s Deep Web Harvesting technology collects content from both the Surface Web and the Deep Web; delivering you more data than a traditional collection technology. To learn more about the Deep Web and the type of content found there download our Understanding the Deep Web whitepaper

  • Harvest Content, Not Links

When BrightPlanet harvests data we harvest all the text-based content on the page and store that content offline rather than just giving you a link to the URL. This gives you a few advantages that include:

    1. Pages that have been harvested and then removed will still have a record
    2. Ability to integrate harvested data with 3rd party analytics
    3. Ability to monitor pages for changes made to content
  • Harvest Content at Big Data Scale

BrightPlanet’s Deep Web Harvester is highly scalable with the capability to issue hundreds of queries (aka searches) to hundreds of websites simultaneously. To read more about the Deep Web Harvester visit here

Solutions for Defining Your Own Data Set

Do you need just monitoring and alerting services?

BrightPlanet’s Deep Web Monitor is the perfect fit for you. Pricing starts at $249/month and allows you to define your sources, use your own keywords, and configure email alerts for new and modified content.

To read more about the Deep Web Monitor click here.

Do you need analytics to go with your harvested data?

BrightPlanet’s Silo Services are a good fit. BrightPlanet’s Silo Services include topic specific repositories of data harvested to your needs. Pricing varies according to the size and scope of each project.

To read more about the Deep Web Silo Services click here.

Next Step

If you are ready to define your own data set, sign up for a free consultation with one of our Data Acquisition Engineers to talk about the deliverable that is right for you.



Photo: bionicteaching