Top data science news this week of Aug 22 – Aug 26

Gartner Hype Cycle For Emerging Technologies, 2016 Adds Blockchain & Machine Learning For First Time

Gartner added 16 new technologies to the Hype Cycle this year, including blockchain, machine learning, general purpose machine intelligence, smart workspace for the first time.  Read more

Facebook’s Artificial Intelligence Research lab releases open source fastText on GitHub

In an effort to classify both accurately and easily, Facebook’s Artificial Intelligence Research (FAIR) lab developed fastText. Today, fastText is going open source so developers can implement its libraries anywhere. Starting today, Facebook’s fastText will be available from their GitHub. Read more.

OpenAI is Using Reddit to Teach An Artificial Intelligence How to Speak

Elon Musk’s artificial intelligence (AI) company OpenAI just received a package that took $2 billion to develop: NVIDIA CEO Jen-Hsun Huang just delivered the first DGX-1 supercomputer to the non-profit organization, which is dedicated to “advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return.” Read more.

How Artificial Intelligence Could Help Diagnose Mental Disorders

In 2015, a team of researchers developed an AI model that correctly predicted which members of a group of young people would develop psychosis—a major feature of schizophrenia—by analyzing transcripts of their speech. Now, Jim Schwoebel, an engineer and CEO of NeuroLex Diagnostics, wants to build on that work to make a tool for primary-care doctors to screen their patients for schizophrenia.  Read more.

Get a ton of traffic, just sell ads, you’ll make a fortune

Over the past few years, I’ve run across many entrepreneurs who have the next “big thing” and their business model is advertising. What these eager entrepreneurs fail to understand is that digital advertising – particularly in the consumer web – has become a commodity business. While many a fortune has been minted in digital advertising on the consumer web, this is the exception, not the rule. For every mega-success story, there are probably a thousand consumer websites that just sputter along, barely covering the cost of running the site.

They key: high-value audience

If you’re reading this, you’re undoubtedly interested in what it takes to make a successful advertising business.  The key is really the audience that you target, it comes down to a high-value audience – that is an audience with a  high lifetime value and hard to target through other means.  I’ll describe this in greater detail below.

Why is the consumer web a commodity business?

First, let’s step back to where the problem lies with the consumer web.  As I’ve already mentioned, advertising on the consumer web is a commodity business. While there are many factors that cause this, two primary forces driving this today are A) over-supply of consumer advertising choices (Google is a huge part of this, but also an abundance of digital media properties/channels for selling ads), and B) the new trend of programmatic ad buying in which automated exchanges make the price paid more efficient, which will continue to drive ad-revenue lower for the website owner / publisher.

What to look for in building a high-value audience

The question therefore becomes, is digital advertising a worthy option for creating a business? Yes. But, the key is to find a niche audience that is valuable to a potential advertiser and is hard to target broadly.

As an entreprenuer starting a new business/product with an advertising business, focus your efforts on a few key attributes:

1. High-lifetime value of a customer

Put yourself in the marketers shoes for a minute – marketer is the “buyer” of your product, i.e.: audience. As a marketer, your basic formula is this: Customer Acquisition Cost + Cost of Product < Price of Product. In simple terms, your customer must pay you more money than what it costs you to produce and market the product.

With consumer products, margins tend to be very thin. This means the profit on the product is low, which means the marketing budget per product is low. Let’s look at a few examples.

Let’s assume you’re doing a simple marketing campaign using some industry averages ($10.87 CPM, .11% click through rate, 3% visit to purchase conversion rate). You purchase a 10,000 impressions campaign – which is small, by the way.

Imagine, as a marketer, you make a product that sells for $20, it costs you $10 to produce that product. Your margin is $10. Your financials for a single sale on your 10,000 impression campaign look like this:

1

Not good.

A better option is to focus on customers who have a high lifetime value – that is, a customer who will spend a lot over their lifetime with you. That could be a single sale; however, it can also be a customer that continues to spend over the course of a few periods, years for example. To illustrate the point, consider a subscription type product where a customer might spend $100 per year for, on average, a total of 5 years. In cases like these, a marketer can afford to spend a lot more money to “acquire” that customer.

The example might look like this:

2

Looks a lot better.

In another example, consider a company that sells iOS App development services. The average cost for an iOS App is $34,000. An iOS Developer / agency can afford to spend a lot of money to reach one customer:

3

What does this all mean? Well, it’s easier for the marketer to buy ads – the margins are there. For you, the publisher, it also means you can afford to charge more for your audience via advertising.

2. Hard to target audience (outside of search keywords)

While high lifetime value is important, there is another dimension to consider – that is, what options does the marketer have to “reach” the audience. Think back to your marketing 101 class, marketing is about targeting the right product with the right message, to the right audience, at the right time.

This isn’t rocket-science, this is what has made Google’s ads so effective – I am a marketer trying to sell wedding dresses, if someone types “Wedding Dress” into google, there is a good chance she is looking to buy a wedding dress. The key? Google has, rather efficiently, brought a prospect to a marketer based on what she is interested in/looking for.

While Google can do this in a multitude of industries/products with keywords, there are many other options for a marketer to reach the “wedding dress” demographic (women between the age of 20 – 35). A few examples might be: fashion blogs, wedding magazines, facebook demographic targeting (plus keywords), bill boards. The demographic is broad, you’re likely – albeit in a less-targeted way – to find prospects with simple demographic cues. As we’ve said above, broad supply of options makes your consumer media property less attractive, more of a commodity.

Thinking of the examples above, people who spend a lot of money on a particular product, the next attribute to consider is “hard to target based on traditional demographics”. Consider the example, again, of someone who wants to build an iOS App. Who might you target? There are a few job titles you can consider – such as marketing managers, but you cant easily turn to mass media to find this person. Therefore, someone who is difficult to target based on standard demograhpics – job title, age, etc – focus on someone who is only targetable based on what they are searching-for/reading. Which means, while there is a broad supply of advertising options, they are likely not as effective as a website that “teaches” people what to look for when hiring an iOS Developer.

In conclusion, digital advertising – particularly mass consumer advertising – has been increasingly commoditized because of great supply of options and downward pressure on what marketers will spend to reach a consumer. If you’re considering creating a startup focused on an advertising model, look to attract a community who A) has a high-lifetime value to the marketer (company) who wants to reach that person and B) is difficult to target based on standard demographics, but is better-targeted by the type of actions they exhibit on the web (ie: searching for specific articles/terms).

Christensen and Raynor’s RPV Framework is summarized in the following article: A Framework for Choosing Organizational Structure and Home. The topic of this article is people – people are part of the R (Resources) in the RPV Framework. People can be one of the most valuable components of the innovation process, particularly when they develop skills in developing disruptive businesses which can then be transported to other areas within the organization. Conversely, people can be one of the weakest links in the innovation process.

Christensen and Raynor have examined innumerable failed efforts to create new-growth businesses. According to their estimates, the efforts failed because the “wrong people” had been chosen to lead the venture. Most managers will immediately conclude that “wrong people” means inadequate skills in the traditional sense of “the right stuff attributes” – such as results oriented, a string of past successes, people skills – this is exactly the type of thinking that led to selecting the wrong person. Common logic would suggest that a string of past successes will predict future success – this is wrong. According to Christensen and Raynor, it is folly to conclude that past success predicts future success, right-stuff thinking gets the categories wrong. Managers should look to circumstance-based attributes, or “schools of experience” criteria to guide the selection process.

Schools of Experience Hiring Method

Professor Morgan McCall articulates a circumstance-based theory which serves as a much more reliable guide for executives who wish to place the “right people” in the “right positions” when launching a new growth businesses. The remainder of this post has been excerpted from The Innovators Solution.

McCall asserts that the management skills and intuition that enable people to succeed in new assignments are shaped through their experiences in previous assignments in their careers. A business unit therefore can be thought of as a school, and the problems that managers have confronted within it constitute the “curriculum” that was offered in that school. The skills that managers can be expected to have and lack, therefore, depend heavily upon which “courses” they did and did not take as they attended various schools of experience.

For instance, a senior manufacturing executive would likely be weak in starting a new plant because his experience is in running a well-tuned plant – he has never encountered the type of problems, which are very different, in starting a new plant.

How to select a candidate based on “Schools of Experience”

In order to be confident that managers have developed the skills required to succeed at a new assignment, managers should examine the types of problems they have wrestled with in the past. It is not as important that managers have succeeded with the problem as it is for them to have wrestled with it and developed the skills and intuition for how to meet the challenge successfully the next time around. We often learn more from our failures than successes. A manager can achieve this in three steps:

Step 1: List the Problems and Challenges

The first step is to list what types of problems or challenges you might expect that the venture might encounter.

Step 2: Which Courses

Using McCall’s Theory as our guide, list the “courses” that we would want members of the team to have taken in earlier career assignments in the school of experience – experiences through which they would have developed the intuition and skill to understand and manage this set of foreseeable problems. This listing of experiences should constitute a “hiring specification” for the senior management team. Rather than specifying a set of right-stuff attributes, the first step specifies the circumstances in which the new team will be asked to manage. The second step matches those circumstances against the problems and challenges with which the managers of the new venture need already to have wrestled.

Step 3: Compare

Our third step would be to compare that set of needed experiences and perspectives with the experiences on the resumes of the candidates. Finding managers who have been appropriately schooled is a critical first step in assembling the capabilities required to succeed.

Source: The Innovators Solution

In The Innovators Solution, Christensen and Raynor propose a method to define what organizations can and cannot accomplish – the RPV Framework. The RPV framework is composed of Resources, Processes, and Values.

Christensen and Raynor propose a decision-making model to help managers of New Growth Businesses assess the correct organizational structure and home for the New Growth Business.

In some cases, when launching a New Growth Businesses, managers should deploy what Business School Professors Kim Clark and Steven Wheelwright call a heavyweight team.

Heavyweight vs Lightweight Teams

When new processes need to be created, it requires a heavyweight team. The term refers to a group of people who are pulled out of their functional organizations and placed in a team structure that allows them to interact over different issues at a different pace and with different organization groups than they habitually could across boundaries of functional organizations. Conversely, lightweight teams are tools to exploit existing processes.

When to use a Heavyweight Team

When there is a well-defined interface between the activities of two different people or organizational groups – meaning that you can clearly specify what each is supposed to deliver, you can measure and verify what they deliver, and there are no unanticipated interdependencies between what one does and what the other must do in response – then those people and groups can interface at arm’s length and do not need to be on the same team.

When these conditions are not met, then all unpredictable interdependencies should be incorporated within the boundaries of a heavyweight team. The team’s external boundary can be drawn where there are modular interfaces.

To be successful, heavyweight teams should be co-located

Source: The Innovators Solution

“A surprising number of innovations fail not because of some fatal technological flaw or because the market isn’t ready, they fail because the responsibility to build these businesses is given to managers or organizations whose capabilities are not up to the task.” – Christensen & Raynor

In The Innovators Solution, Christensen and Raynor propose a method to define what organizations can and cannot accomplish – the RPV Framework. The RPV framework is composed of Resources, Processes, and Values.

Christensen and Raynor have developed a decision-making model, the purpose of which is to help managers determine the proper organizational structure and “home” for the new-growth business. Their model is below.

A Framework for Choosing Organizational Structure and Home

framework_for_finding_organizational_structure

Axes

Fit with Organization’s Processes

The left vertical axis measures the extent to which the existing processes – the patterns of interaction, communication, coordination, and decision making current used in the organization – are the ones that will get the new job done effectively.

Fit with Organization’s Values

The lower horizontal axis asks managers to asses whether the organization’s values will allocate to the new initiative the resources it will need in order to become successful. If there is a poor fit, then the mainstream organization’s values will accord low priority to the project; that is, the project is potentially disruptive relative to its business model.

Position of Responsible Commercial Structure

The upper horizontal axis captures (on a continuum) the level of autonomy needed by an organizational unit attempting to exploit an innovation. For disruptive innovations, setting up an autonomous organization to develop and commercialize the venture will be absolutely essential to its success. At the other extreme, however if there is a strong sustaining fit, then the manager can expect that the energy and resources of the mainstream organization will coalesce behind it because the project is sustaining.

Structure of Development Team

The right vertical axis maps three types of organizational sturcutres that can be used to either exploit or overcome existing processes.

How to Use the Model

Overview

Using the model requires the manager to assess the organizations processes and values. Quite simply, the bottom and left sides of the model are used for “diagnosis”, the top and right sides of the model provide the “solution”.

First, evaluate the processes required to launch and scale the New Growth Business. If the New Growth Business requires new processes (poor fit), then it will require region A or C. If the New Growth Business can utilize existing processes (good fit), then it will require region B or D.

Next, evaluate the organizations values – according to Christensen and Raynor’s definition of values – and select the appropriate quadrant.

Region A

Region A depicts a situation in which a manager is faced with a breakthrough but sustaining technological change. It fits the organization’s values, but it presents the organization with different types of problems to solve and therefore requires new types of interaction and coordination among groups and individuals.

Region B

In region B, where the project fits the company’s processes as well as its values, the new venture can easily be developed by coordinating across functional boundaries within the existing organization.

Region C

Region C denotes a disruptive technological change that fits neither the organization’s existing processes nor its values. To ensure success in such instances, the managers should create an autonomous organization.

Region D

Region D typifies projects in which products or services similar to those in the mainstream need to be sold within a fundamentally lower-overhead business model. These ventures can leverage the main organization’s logistics management processes, but they need very different budgeting, management, and profit and loss profiles.

Caution

In using this model, it is important to remember that disruption is a relative term. What is disruptive to one company might have a sustaining impact on another.

Sources
* The Innovators Solution

A product architecture determines its constituent components and subsystems and defines how they must interact – fit and work together – in order to achieve targeted functionality. The place where any two components fit together is called an interface. Interfaces exist within a product, as well as between stages in the value chain.

An architecture is interdependent at an interface if one part cannot be created independently of the other part – if the way one is designed and made depends on the way the other is being designed and made. When there is an interface across which there are unpredictable interdependencies, then the same organization must simultaneously develop both of the components if it hopes to develop either component.

In contrast, a modular interface is a clean one in which there are no unpredictable interdependencies across components or stages of the value chain. Modular architectures optimize flexibility, but because they require tight specification, the give engineers fewer degrees of freedom in design.

When the functionality and reliability of a product are not good enough to meet customers’ needs, then the companies that will enjoy significant competitive advantage are those whose product architectures are proprietary and that are integrated across the performance limiting interfaces in the value chain.

When functionality and reliability become more than adequate, so that speed and responsiveness are the dimensions of competition that are not now good enough, then the opposite is true. A population of nonintegrated specialized companies whose rules of interaction are defined by modular architectures and industry standards holds the upper hand.

At the beginning of a wave of new-market disruption, the companies that initially will be the most successful will be integrated firms whose architectures are proprietary because the product isn’t yet good enough. After a few years of success in performance improvement, those disruptive pioneers themselves become susceptible to hybrid disruption by a faster and more flexible population of nonintegrated companies whose focus gives them lower overhead costs.

Source: Innovators Solution

Christensen observed that when upstart entrants attacked successful incumbents by adopting the incumbents’ models and technological solutions — what he called a “sustaining” strategy — they tended to fail.

In contrast, entrants tended to succeed by combining a business model tailored to the needs of a relatively less attractive market — the entrants’ foothold — with an ability to improve their original solutions in ways that allowed them to provide superior performance that incumbents were unable to replicate — the upmarket march. Christensen called the union of these two elements a “disruptive” strategy.

* Source: Deloitte

Why Market?

Some Lean practitioners have abandoned planning altogether – citing, research and planning is a form of waste and doing is better than planning. In many cases doing is better than planning; however, in this case, planning is better than doing. Before you take the leap of launching a new business or product, your first step should be a simple market analysis.

In the startup world, there have been countless debates as to whether it is “team”, “product” or “market”. In one of the most-quoted blogs posts on the topic, Marc Andreessen says:

In a terrible market, you can have the best product in the world and an absolutely killer team, and it doesn’t matter — you’re going to fail.

You’ll break your pick for years trying to find customers who don’t exist for your marvelous product, and your wonderful team will eventually get demoralized and quit, and your startup will die.

Andreessen goes on to quote Andy Rachleff’s Law of Startup Success:

The #1 company-killer is lack of market.

When a great team meets a lousy market, market wins.

When a lousy team meets a great market, market wins.

When a great team meets a great market, something special happens.

The point is this, the most important thing you can do is to pick a great market, so do some simple market research before you start working on your “great idea”.

Unfortunately, there is no single tried and true method for researching every market – markets are unique, replete with their own nuances and attributes. However, at a minimum, your market analysis should include a few simple attributes: market size and growth rate, competition, and market trends.

How to Evaluate the Market

Market Size

For those of you who are looking for more practical advice, there is a great post on Ultra Light Startups that outlines multiple methods of how to size a potential market.

In The Four Steps to the Epiphany, Steve Blank provides an easy-to-follow method of creating a top-down and bottom-up approach market size hypothesis.

Top-down

A top-down estimate starts with a total market size number then assumes your company can sell-to a percentage (usually) of that market. Here is an example from Deviantbits:

  • We want to sell internet access in China
  • There are 1.3 billion people
  • 1 percent want internet access
  • We’ll get 10 percent of that potential audience
  • Each account will yield $240 per year
  • 1.3 billion people × 1% addressable market × 10% success rate ×$240/customer = $312 million

Where can you find the information to compile a top-down estimate? You can find market statistics in: industry-analyst reports, market­
research reports, competitors’ press releases, university libraries, and government statistical reports.

Bottom-up

While top-down has its place, a bottom-up estimate is more appropriate for a small startup company/venture because it is based on sales/performance data specific to your situation – such as your marketing budget, how many customers you can serve, etc.

In other words, a bottom-up approach makes assumptions about what you can achieve using your specific sales/marketing tactics.

Again, another example from Deviantbits:

  • Each salesperson can make ten phone sales calls a day that get through to a prospect
  • There are 240 working days per year
  • Five percent of the sales calls will convert within six months
  • Each successful sale will bring $240 worth of business
  • We can bring on board five salespeople
  • Ten calls/day × 240 days/year × 5% success rate × $240/sale × 5 salespeople = $144,000 in sales in the first year

What if you don’t have a sales team? Google Adwords keyword tool is a great source of bottom-up forecasting. For example, you could estimate how many visits you can drive to your website x conversion rate x average order value.

The side benefit of the bottom-up approach – in good ole’ lean fashion – is that your assumptions are testable.

Growth Rate

Now that you’ve sized your market, you need to look at the market growth rate. Understanding the growth rate will help you understand the potential of the business as well as the lifecycle stage of the business – more on that to come.

Depending on the market you’ve selected, you can typically find a growth rate through research reports, blogs, press releases, etc. In other words, start with google.

In some cases you’ll find a growth rate percentage. In other cases you may need to calculate it based on size statistics. The following is a simple example of how to calculate growth rate:

  • In 2012, the market size was 200mm
  • In 2013, the market size was 240mm
  • The market growth rate is: [(240-200)/200] x 100 = 20%

How does this relate to the potential? You can use this to forecast the growth in revenue:

  • Year 1: you might expect sales at 200mm
  • Year 2: 240mm (200 x 1.20)
  • Year 3: 288mm (240 x 1.2)

Caveat: in Lean fashion, this is an assumption, not a “given”, this is a benchmark to help you understand the scale and scope of the business.

In addition to using the growth rate for forecasting, you can also use it as a health benchmark. For instance, if your sales growth is greater than or equal to market growth, your firm is comparatively healthy. If, however, your company’s growth in sales is less than market growth, it is very likely your firm is in competitive trouble, especially if this is not your strategy (Frost & Sullivan).

Lifecycle Stage

The market growth rate is a key indication of the product’s stage in the product life cycle. A high growth rate will usually indicate the market is in the growth phase, where growth is high and saturation is low. A lower, more-stable growth rate indicates product maturation and, of course, a negative market growth rate indicates the product decline stage (Frost & Sullivan).

As a startup company, your goal is to focus on introduction and early growth phase markets/products.

Phase Characteristics
Introduction
  • Growth: A low growth rate, small market size
  • Market potential: Uncertain
  • Competition: Few competitors/little competition
  • Barriers to entry: low
Growth
  • Growth: dramatic/increasing growth (size/no. of customers)
  • Market potential: Strong
  • Competition: Moderate, increasing competitors/competition
  • Barriers to entry: increasing
Maturity
  • Growth: Positive but decreasing growth rate (size/no. of customers)
  • Market potential: moderate
  • Competition: High, many competitors, consolidation starts occurring
  • Barriers to entry: high
Decline
  • Growth: Negative, declining growth rate
  • Market potential: low
  • Competition: Decreases, firms exit market
  • Barriers to entry: high

Tip: Ideally, focus on the first stages: introduction and growth.

Trends

Market trends is the first of the two more-subjective analysis criteria. In terms of trends, research the trends in the market – you will want to identify the top 2-3 trends that are likely to impact your market.

For the sake of simplicity, look at products and markets (see the Ansoff Matrix and focus on macro and micro trends.

  • Markets: (customers) what are the big trends that will affect your customers – changes in how people buy, economic trends, etc.
  • Products: what are the big trends that affect your products – competitors actions, changing technologies, etc.
  • Macro: broad changes – you can read these on blogs, in the mainstream news, etc.
  • Micro: talk to “real people” – what are customers saying, how do people interact with products, etc.

Competition

Finally, you will want to research the top competitors – which includes alternatives and substitutes – to your product or service. To illustrate the difference between competitor and substitute, consider the example of a Nikon digital camera:

  • Alternative: Canon digital camera, same product/category
  • Substitute: iPhone, same feature/benefits/different category

You should be familiar with:

  • Who is the competitor
  • What do they offer (features/benefits)
  • Their core value proposition – in other words, why do customers chose their product over the alternatives

The following is an example competitor analysis:

Nikon Digital Camera
Competitor Type Core offering Main Value Proposition
Nikon Quality compact digital camera at a competitive price. Lenses – faster, better for “action” shots
Canon Alternative Quality compact digital camera at a competitive price. Comparable features to Nikon. Lenses – better for capturing detail (people/scenery)
iPhone Substitute 8 megapixel camera, multi-functional device Multi-functional device, music, camera, phone, entertainment in a single device

This analysis will inform your competitive advantage and differentiation strategy.

Note: I have limited expertise in Canon vs Nikon, so this is more illustrative than accurate.

Conclusion

In conclusion, your final market analysis should posses the following details:

Quantitative

  • Size: target above 100mm size
  • Growth rate (Compare the market growth rate to the economic growth rate. If your category is growing faster than the overall economy, you’re on the right track. Identify where you’re at in the product life cycle )
  • What phase of lifecycle (should be first half)

Qualitative

  • Top 3-5 trends affecting your product/market – positive or negative
  • Competition: list your top competitors and/or main substitutes

Sources: Medium.com, Frost and Sullivan

Executives must answer three sets of questions to determine whether an idea has disruptive potential. The first set explores whether the idea can become a new-market disruption. For this to happen, answers to at least one and generally both of two questions must be positive:

  • Is there a large population of people who historically have not had the money, equipment, or skill to do this thing for themselves, and as a result have gone without it altogether or have needed to pay someone with more expertise to do it for them?
  • To use the product or service, do customers need to go to an inconvenient, centralized location?

If the technology can be developed so that a large population of less skilled or less affluent people can begin owning and using, in a more convenient context, something that historically was available only to more skilled or more affluent people in a centralized, inconvenient location, then there is potential for converting the idea into a new market disruption.

The second set of questions explores the potential for a low-end disruption. This is possible if the answer is yes to two questions:

  • Are there customers at the low end of the market who would be happy to purchase a product with less performance if they could get it at a lower price?
  • Can we create a business model that enables us to earn attractive profits at the discount prices required to win the business of these overserved customers at the low end?

Often, the innovations that enable low-end disruption are improvements that reduce overhead costs, enabling a company to earn attractive returns on lower gross margins, coupled with improvements in manufacturing or business processes that turn assets faster.

Once an innovation passes the new-market or low-end test, there is still a third critical question:

  • Is the innovation disruptive to all of the significant incumbent firms in the industry? If it appears to be sustaining to one or more significant players in the industry, then the odds will be stacked in that firm’s favour, and the entrant is unlikely to win.

If an idea fails what Christensen and Raynor refer to as these litmus tests, then it cannot be shaped into a disruption.

Source: Innovator’s Solution

Landing Page Test

Set up a landing page and get people to perform a specific action

Has three variables:

  • How good is your acquisition method (say, google Adword)
  • How good are your message positioning and design
  • How good is your value proposition?

Concierge Test

Testing by doing, substituting a human/manual process instead of automating (ie: creating a product)

Two things it can answer:

  • Is the problem solvable with a product?
  • Are customers willing to substitute human service with a product?

Wizard of OZ Test

Similar do concierge, build a front end of your system but no blackened. Humans do the back end manually.

Example: Zappos

  • The riskiest question was, “will people buy shoes online”. Rather than raising millions, building a infrastructure and buying inventory, the founder went to Foot Locker and took pictures of their inventory. He put photos of the shoes online. Every time someone placed an order, he went to Foot Locker purchased the shoes and mailed them to the buyer.

Read more examples

Smoke test

Done with marketing materials. Customers are given the opportunity to preorder a product that has not yet been built. A smoke test measures only one thing: whether customers are interested in trying a product.

Similar test is crowd funding test, test demand by getting crowd funding for pre orders.