How You Can Mine Market Data.. for Free

Gone were the days when government data is off-limits. Data can be very expensive. That’s why before some companies mine these data, packaged them and sell it for boatload of money to their customers. Nothing wrong with that, but if you are just starting up and you need to do market research, there are places where you can access data for – free. 

Depending on the type of market data you’re looking for, you can explore the world. What you want to do is you to narrow down the field specifics. And then drill down or expand from there.

I’m going to list of some websites, where you can mine data for free. Websites from the US government, International organizations, such as the World Bank, United Nations, OECD (Organization for Economic Co-operation and Development), other countries government sites, state and local governments, trade associations, NGOs and more. Technically, you can pretty much get everything online – these days.


I heard in a seminar that NOAA (Natl. Oceanic and Atmospheric Administration – part of Dept. of Commerce) opened up their data a while back. In essence, their data powered the Weather Channel and some other weather-related sites. HHS (Dept. of Human Health and Services) have a number of key sites where you can get data related to human health, for example, where you can find different health plans available to you down to the area where you live. Sometime in the summer of this year, they’re planning to release some 1,000+ health indicators that you can literally nail it down to pricing. The idea is to give consumer the access to information so they can decide what’s good for them and make the cos. compete for your business.


This uber site is where you can track opportunities, see where they money is going, etc. The site is linked to, where you can information about contracts;, where you can get info what type of grants available, and most important – if you are qualify;, to find the right loans from business loans, business industrial loans, agriculture, education, housing, veteran loans, etc.; and, with focused specifically targeted to small businesses. 


The Commerce Department has 12 bureaus including Census Bureau, Minority Business Development, NOAA, NTIA, NIST, USPTO (Patent).

Most business data is available through the Census Bureau. The census have a list of economic indicators, that you might have heard via the news. Let’s say you are in construction business, you want to know the trend. These are sample of the list of data available to you: housing vacancies and homeownership, new home sales, housing starts/ building permits. In addition to economic indicators, it also offers access to economic census that is released every 5 years, other economic programs that is primarily national. 

Digging in deeper, you’ll find market data by sector: economy wide, construction, governments, international trade, manufacturing, retail, services, wholesale, and other sectors (agriculture, mining and utilities). 

You might want to bookmark this site for business data you can use:

Determine which data you need, you then can look at data, i.e. the services sector and then get the industry snapshots that will show the size of market, and which market is growing within the industry.


Want to know which market is trending up? NAICS site is the place. There are 11 sectors that fall into the category within NAICS (North American Industry Classification System). If you’re thinking about starting up a business, this is the place where you want to spend a lot of time digging in market data. You can see statistics of U.S. businesses down to the number of companies, employees, payroll, revenue by employment-size. Half of the businesses in the U.S. is small business under 4 employees. However, the big companies – those over 500 employees – employs half of the U.S. workforce. Make sense now, why there are more incentives for the big businesses than small biz. It’s the headcount.

You can find data for most industries, via NAICS. This is where you can drill down to the n-th degree of data. For example, the industry snapshots for Professional, Scientific and Technical services. 


When you see the button more, click on it and then it will expand to list of industries within the sector, from legal services to accounting, architectural services, management consulting to other professionals not listed there. You can drill down data to the type of industries within the type of services. Within legal services, there are offices of lawyers, other legal services, title abstracts and settlement offices, and others. 


The point is, when you have to justify whether the industry you’re trying to enter shows growth trend, flat or downhill – the info is there. You can make informed decision.

Green industries like environmental consulting, and other scientific, technical services are growing industries, according to the data. Revenues of environmental consulting grew 49% from 2002 to 2007. Highly-specialized businesses like programming, computer facilities management, etc., show increasing trend. This is a good sign. Green is good business.


Starting up? Head off to this site. Resources include general statistics, demographics, consumer stats, economic indicators, employment stats, income and earning, trade, specific industries as well as state and local info. It actually link directly to other government sites, i.e. Census, Bureau Labor Statistics. 

The bonus: there’s a community site over at this site, where you can discuss about business issues or ask questions. It’s a controlled community. You can’t promote your business at the site, otherwise they’ll kick you out. You can only network.


Want to know how to get started, what industries, do market research, find opportunities, what kind of contracts available through multilateral banks, banks like African Development Bank, Asian Development Bank, Inter-American Bank, World Bank and European Bank for Reconstruction and Development. 


The World Bank

Recently, the WB released some 2,000 indicators you can access for free that were originally off limits. It’s called the World Bank’s Open Data. They have a catalog of data that listed the 2,000 indicators available in Excel, CSV, and API. The type of data you can access there includes health, education, poverty, countries profiles, economic development, environment. 

In fact, WB is going to launch ‘apps for development’ competition. Later in 2011, whatever data you’re looking for.. there’s a app for that. 


The image above shows the two areas, where we can do something about: education and environment. 

UN (United Nations)

Within the UN site, you can get access to database on economic development from around the world. Within the UN bodies – for those of us who’s in the business of sustainability – UN Global Compact is the place to go. 

OECD (Organization for Economic Co-operation and Development)

In other words, the organization for developed countries. OECD has selection of data available with no subscription. Some of their data is by subscription. However, the ones available with no subscription, is a long list. It’s called OECD.Stat.Extract. You can access by keyword, theme, query, and country. 


European Wind Energy Association is a great place to see the progress for wind energy. There are reports, factsheets, position papers, and free magazine you can subscribe to. 

Okay, I think you get the idea. That you don’t need to scoop big bucks to get market data. We are at the time, when small and medium-sized businesses can compete with big businesses. So, there’s no reason that you cannot find access to free data, all you need is time, a little bit of motivation and perseverance. Almost everything is now available online. Some places might require you to sign up, but hey, it’s worth it. 

By the way, if you know websites where you can get data for free or minimal cost and you think the information can help others, let me know in the comment section below.


Author: Dewita

Co-founder Ecotwist Labs.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s