Our Technology

Harvest

We’ve developed a patented Deep Web Harvester that allows us to harvest data from web pages using pre-existing harvest types. BrightPlanet’s Deep Web Harvester contains the following harvests. All of BrightPlanet’s harvests are highly scalable and allow for the harvesting and collection of thousands of web pages.

• Site Harvests use the technique of link crawling to harvest data. Starting URLs are identified and the harvester crawls based off of links on the web page.
• Deep Web Harvests automate queries directly into web forms to find and harvest Deep Web data.
• RSS Harvests ingest results from RSS feeds to help you stay on top of new content published to news and blog sites.
• E-mail Harvests automate the harvesting of text content from e-mails in an e-mail account the harvester has access to.
• Twitter Harvests utilizing the Twitter Search API to harvest tweet data.
• Facebook Harvests operate using the Facebook Open Graph API.
• Scripted Harvests provide more control over harvests through a scripted language to collect data when websites require custom interactions, such as logging in with a username and password.

Curate

Once the Deep Web Harvester identifies the page to harvest, the curation process begins. The curation process involves structuring the data and converting it from a web page. The process begins with extracting all the text from the page and placing it into a completely unstructured format.

The data is now ready to structure. We structure data using a process called entity extraction. Entity extraction involves identifying key terms within a web page. These typically involve the names of people, companies, and places. Extraction uses a rule-based engine, which means we can also customize entities for each customer to tag the names of your products, and other terms important to you.

Develop Insights

Now that the data is harvested and curated, it’s ready to then begin asking the data questions. BrightPlanet customers typically interact with harvested data one of three ways.

1. Search Dashboard – BrightPlanet’s Search Dashboard gives your team direct access to a full text, searchable repository of all your data harvested in one place.
2. API Access – BrightPlanet’s REST API gives you direct access to data to ingest the data for your own use and analytics internally.
3. Custom Data Visualization – BrightPlanet uses open sources and proprietary data visualization tools to help better analyze and develop insights from the data.