The text analytics market, which has been predicted to be worth US$21.7bn by 2025, is falling headfirst into Gartner’s well-known trough of disillusionment by failing to deliver real organisational value and meet user expectations. The might of marketing has duped clients with hyped promises of illusive actionable insights delivered through fast, sexy interfaces, yet the industry is not delivering the value it promises. This paper explores the reasons behind the failure to deliver this expected value. It will define the terms ‘value’, ‘insight’ and ‘actionable insight’. It will use these definitions to identify where and why industry practice fails to meet these fundamental expectations. A short case study is included to provide an example of how emotion analytics of consumer-generated unstructured text data can help deliver meaningful and genuinely actionable insights.

Download White Paper here

This paper first appeared in the peer-reviewed journal Applied Marketing Analytics, Vol.5 #4 2020. ISSN:2054-7544

Title: Why human involvement is still required to move text analytics technologies leveraged with artificial intelligence from the trough of disillusionment to the plateau of productivity

Introduction

The perceived need to extract value from the exponentially growing volume of unstructured text on the internet and within organisations is contributing to the rapid and persistent growth of the text analytics market, which is forecast to grow at a compound annual growth rate (CAGR) of 27.4 per cent to reach US$21.7bn by 2025.¹

This market forecast combined with the rapid technical advancement in various artificial intelligence (AI) technologies, specifically natural language processing (NLP) and machine learning (ML), has removed a barrier to entry to the market. These conditions have subsequently generated a plethora of start-ups and given a technical capability to existing businesses with a route to market for text analytics.

The hype

Among experts, however, AI is recognised as being ‘poor at dealing with unknown and unstructured spaces’,² including unstructured text. The result is a hungry industry that has set high expectations, promising ‘illusive actionable insights’,³ but that is failing to deliver any real value.

Gartner’s Hype Cycle for Artificial Intelligence places NLP and ML at the top of the hype cycle⁴ about to fall unceremoniously into the trough of disillusionment. Gartner predicts it will take from five to ten years for NLP technology to reach the plateau of productivity.

We shouldn’t be at all surprised, as this hype cycle aligns to the innovation S-curve (Figure 1). The birth of a new innovation is an exciting time, with lots of attention and inevitable hype.  It’s typically some time (Slope of enlightenment) before the innovative step gets to the growth stage (Plateau of productivity). Many casualties will fall by the wayside on this innovation journey.

Figure 1: The S-curve and hype cycle

Technology

In the field of text analytics, NLP and ML are used to classify language, identify themes and sentiment within text datasets. Both are used to identify and categorise qualitative variables within unstructured text, thereby structuring it and enabling further analysis via quantitative methods.

While both NLP and ML technologies can decipher meaning within text, their power is limited. A text analytics platform that relies too heavily upon these technologies will be unable to structure unstructured text data sufficiently well to ensure further analysis can deliver relevant, or even correct, insights.

An example would be the topic modelling functionality of NLP. It automatically identifies topics (patterns of words) presented within the text. The technology may identify the word ‘expensive’ or ‘pricey’ as being used a lot. Unless someone teaches the machine, in this domain of activity and context, that these two words mean the same thing, it will find two topics. Thus, the frequency of use is diluted to such an extent that it is possible to completely lose the quantitative significance of this topic.

Fundamentally, the machine has yet to be taught how to recognise and therefore understand anything but the most basic of the myriad aspects of human language. The process of teaching the machine to identify the meaning, context and relevance of words in multiple domains of human activity is underway but a huge challenge.

Gartner’s long slope of enlightenment (5–10 years for NLP and ML) goes some way to explain the journey the industry is on. Inevitably, some organisations will be further along the journey than others. Those that are further along will be able to add commercial value, while those who lag behind will drown in their marketing department’s hype.

The dominant approach is to enable the machine to learn by providing a vast database of millions, or billions,⁵ of professionally annotated training examples from which it can learn to identify topics and thus predict the meaning in an extract of text.

The extreme quantities of such comments required results in major players on the AI landscape employing tens of thousands of graduates to tag multiple millions of text examples.⁶ This is time and resource heavy. While this is not the only approach, what remains true is that there really is no option other than for smart, experienced people to teach the machine.

Bag of Words (BoW) has been around since 1954.⁷ Its somewhat derogatory name does not give justice to either its potential or complexity. Indeed, it alludes to simplicity. The BoW is simply another way to teach the machine and it is equally on its own journey. At the start of the journey it is a list of basic words, while the destination is thousands of bags of words representing a thing, or the meaning of something. Each word is human-curated from the everyday language that people use in particular domains of activity. Each bag containing multiple words or phrases, each word or phrase wrapped in hundreds of conditionalities (rules) to ensure context accuracy. The BoWs themselves are arranged in domain-specific ontologies that facilitate the identification of multi-dimensional relationships. Once again, some organisations using the BoW mechanism are further along the journey than others, able to deliver accordant increases in meaningful value to their clients.

Narrative data veracity

Narrative data can be notionally split into three types:

• Solicited by a survey: When people are asked for feedback on a topic, they are often inclined to either gift, game, guard or guess⁸ their answers, none of which is useful.⁹ Solicited narrative often lacks veracity and richness. Comments are often very short or non-existent. These days, survey-sick customers are bombarded by cheap-to-run, online surveys. To encourage their participation, customers often are rewarded with a gift or monetary remuneration; both greatly increase the likelihood of poor quality and untrustworthy responses.
• Unsolicited feedback from emotionally motivated consumers who have decided to give unbidden time and effort to voicing their experiences: In isolation this might be considered anecdotal. In enough quantities and in alignment with the business question, this should be considered as a very relevant and rich source of data — an evidence base that should not be ignored. What is more, online reviews provide an excellent opportunity to analyse competitors’ products and services.
• Conversational, e-mail traffic, chatbots, call centre voice recordings: This narrative contains real frontline experiences full of ‘why’ information. This kind of data is highly valuable and pertinent to any improvement activity.

Insight

All narrative data will contain descriptions of the qualities of products or the service offering, hence the term ‘qualitative’. Often the narrative will explain why the experience or product was good, bad or indifferent (sentimental values) and sometimes how they feel about it (emotional values). An insight is a deep understanding of a subject matter. This clarity and understanding dictates the need for descriptions of qualities, sentiment, emotion and ‘why’ information to be available for analysis. How else is one going to gain a deep understanding? Digitally analysing volumes of narrative text rich in all these elements goes some way to explain why the industry alludes to insight at every opportunity, as reading even a single comment can be insightful.

Actionable insight

Actionable insight, however, is a completely different beast, as highlighted by Brent Dykes when he stated:

‘Every analytics or business intelligence solution promises to unlock a tidal wave of them [actionable insights] for your business — maybe even in real time if you’re lucky … While the promise of actionable insights is alluring, I’m concerned that the phrase is fast becoming an empty buzzword as it is being overly misused by technology marketers. Often what is really being offered by many analytics solutions is just more data or information — not insights’. ¹⁰

Any customer looking to procure a business intelligence platform or text analytics solution needs to perform their due diligence prior to purchase. To avoid joining a growing cohort of businesses that are wallowing around in the ‘trough of disillusionment’, studying Table 1 will provide insight into, or a deep understanding of, an actionable insight. This will help to steer clear of the hype, provide a good background to challenge suppliers and increase the chances of picking a supplier that is further along the slope of enlightenment and thus more likely to deliver real commercial value.

Table 1: Pansensic’s seven vital attributes of an actionable insight

Above all, actioning an insight needs a human in the loop. Decisions should only be actioned based on insights from multiple sources, intuition and experience and cognitive prediction. Only a human brain can make a proper informed decision that bears all things in mind.

Commercial value

Extracting value from text data demands a clear understanding what constitutes value. Peter Drucker’s insight defines this very well: ‘Marketing and innovation make money. Everything else is a cost.’ ¹¹

Thus, identifying insights from rich narrative text that specifically helps with these two business functions will provide the most value. However, these insights are like needles in the big narrative data haystack. To find them, the text analytics supplier must know exactly what they are looking for and devise a way of finding them. One way of finding these valuable marketing and innovation insights within narrative data is with the use of emotion analytics.

Sentiment analysis and emotion analytics

In 1932, psychologist Rensis Likert devised the Likert psychometric scale to measure attitude and opinion via a questionnaire about specific elements of a product or service. This manner of limiting responses to a choice of opinions between very negative and very positive was arguably the genesis of sentiment analysis. Change the words to measure satisfaction and one could equally state this was the start of emotion analytics. Eighty-eight years later, similar questionnaires are answered by tapping on a row of smiley-faced, or otherwise, buttons or selecting a star rating. This tried and tested approach can provide a set of purely quantitative metrics. However, if there is no collection of the narrative whereby the consumer can explain why they gave that score, then the value is very limited. The fact that 80 per cent of customers are very satisfied with a product does not provide any insight into why 20 per cent are not happy. Identifying opportunities for improvement or innovation through the analysis of the qualities described in any accompanying narrative is fundamental to the extraction of value from surveying customers; however, many survey vendors do not yet have this capability. This will surely change very quickly in the coming years.

Most sentiment analysis tools are based on BoWs but called NLP libraries. The proliferation of sentiment analysis started in around 2005¹².Essentially, the tool classifies the text surrounding a topic into either positive or negative sentiment (opinion).

Sentiment analysis is a sensitive tool as most passages of text will contain some sentiment. This makes it the go-to tool for most text analytic vendors. Their offerings identify topics of conversation within the narrative and give the topic a sentiment analysis score based on the positivity or negativity of words in close proximity to the topic. This is how it should work and due to its aforementioned sensitivity, it is a reliable high-level metric.

In the context of text analytics, emotion analytics is similar to sentiment analysis in that it is most likely to exist as multiple BoWs. Each emotion to be identified, be it positive or negative, will have its own BoW. Like sentiment analysis, these BoWs classify the text surrounding a topic by the emotions present. Emotion is essentially a subset of sentiment. There is typically much less emotion than sentiment to be found in consumer-generated text datasets. However, the fact there are over 3,178 emojis¹³ in existence, many of which represent highly varied and nuanced states of emotion, illustrates how people require the ability to communicate their feelings.

To illustrate the difference between sentiment analysis and emotion analytics, consider the following example. Since 1993, I have been an ardent fan of Apple laptops, delighted with their performance and intuitive operating system. In the context of the Net Promoter Score (NPS), I have most definitely been a promoter. Most of my comments towards the attributes of the product would have ranked as positive. My opinion (sentiment analysis) of them today is that they are probably still the best, but not quite as superior as they were a few years ago. However, I am no longer a promoter of the Apple laptop. Indeed, I jumped ship and purchased a Microsoft Windows laptop a few months ago. Why? Apple laptops have gone from having multiple ports for peripherals a few years ago to just two ports today — one of which is needed for power! Having to purchase additional adaptors to increase the number of ports available to cope with the various scenarios of everyday life is plain frustrating. But this is not all: Apple makes me angry. Apple machines refuse to work with adapters that have not been manufactured by Apple. This forces users to buy Apple-branded products, which are very expensive (and still have too few ports), and to drag around an additional bag of paraphernalia. I want my ports back as I need them to connect to the real world. My HP laptop has ten ports and was less than half the price of a similar spec Apple laptop.

The best sentiment analysis can do here is to identify some positive and negative opinion towards attributes of the products (see Figure 4). Emotion analytics, by contrast, can identify a customer who was once delighted with their product but is now frustrated and angry. Emotions lead to behaviours, and in this real example, my behaviour was to drop Apple after 27 years — a decision I do not regret. By referring to this experience I am now a fully-fledged Apple detractor and as Jeff Bezos said: you make customers unhappy in the physical world, they might each tell six friends. If you make customers unhappy on the internet, they can each tell 6,000 friends’.¹⁴

Apple, are you listening?

People act on emotions not opinions

While sentiment analysis can deliver high-level analysis, it does not break down the analysis to a level that gives granularity and a specific advantage.

The influence of emotions on behaviour has been theorised about for a millennium.¹⁵ Evidence suggests they are states of feeling that result in physical and psychological changes that directly influence behaviour and are often the driving force behind positive and negative motivation.¹⁶

As people’s behaviours are driven by what they are feeling, finding these emotions is the smart way to find improvement innovation and marketing insights and thus extract the value from text data. They are direct and reliable indicators of genuine behavioural intent.

Analysis of emotion within text data aims to achieve a comprehensive understanding (insight) of why consumers behave the way they do by identifying the specific emotion and then defining the drivers of that emotion. In this way it is possible to understand what is causing specific decisions and motivations.

At the time of writing, the Pansensic HTA platform isolates and measures 11 core emotions (see below). However, experience points to five key emotions that lead to the richest improvement, innovation and marketing opportunities:

• Desire — used to identify the wants and needs of the author: ‘Desire’ is predominantly a future-state emotion that exists due to a perceived deficiency. Therefore, desire is classified as a negative emotion. Higher levels of desire exist in detractors as what they want is to remove frustration.
• Frustration — used to identify what needs improving: Frustration is a negative emotion and part of everyone’s personal feedback system telling them something is not as good as it should be. Identifying frustration and the drivers of it therefore makes for the perfect product, service or organisation sensor. People are typically objective when describing their frustrations. Frustration levels are very high with detractors.
• Anger — used to identify what needs improving urgently: Anger is a very negative emotion but is less useful than frustration as people tend to be less objective. Feedback becomes more personal, even aggressive, but the drivers behind anger are still instructive. Anger levels are very high with detractors.
• Delight — used to identify what needs propagating: Delight is a very positive emotion, consumer delight being a lofty goal for any product or service. Delight levels are highest with promoters.
• Excitement — used to identify what needs propagating: Excitement is a very positive emotion. Products and services that excite consumers are very favourable, although it can be challenging to introduce excitement into more mundane products or services. Excitement levels are highest in promoters.

Benchmarked against the seven vital attributes of an actionable insight (see Table 1), insights that contain these five emotions possess both advantage and granularity. It is this extra level of specific granularity (finer filtering) within emotion analytics — the ability to dive into the data to discover exactly what is causing the emotions and subsequent behaviour — that gives it a step-change advantage over sentiment analysis when used as an improvement and innovation tool.

So, sentiment analysis is limited to identifying only one level of opinion, as defined by positive, neutral or negative results, while emotion analytics has the capability to define positive or negative groups of emotions by collating the results across a subset of individual emotions grouped into positive or negative. These subsets of emotions can then be analysed to a lower level of granularity.

Emotion analytics (as criteria for Pansensic’s Hybrid Text Analytics platform) can be summarised as follows:

• positive: five positive emotions: love, delight, happiness, excitement, surprise; and
• negative: six negative emotions: frustration, anger, fear, shock, sadness, desire.

Aligning emotion analytics with NPS

When one compares star ratings or NPS with the emotion positivity contained within the text of reviews (otherwise thought of as consumer stories) one obtains a very strong correlation (Figure 2). This is hardly surprising. The star rating is a user-estimated quantitative score of positivity. The text that accompanies the score is the qualitative description of the reasoning behind it. Therefore, both are measures of the same thing.

Figure 2: Correlation graph of emotion positivity against star-rating scores

The significance of this is that it is now possible to track the various distributions of specific emotions against a star rating and NPS. Furthermore, one can define the dominant emotions exhibited by the three levels of consumer loyalty (detractor, passive, promoter) to a brand, product or service. Promoters talk about products and services meeting needs while passives and detractors talk about products and services not meeting needs. The breakdown of consumer levels, score ratings, and associated emotions are listed below:

• Detractors — score 1–6/10 (dominant emotions are ‘frustration’, ‘anger’ and ‘desire’).
• Passives — score 7–8/10 (dominant emotions are ‘happiness’ and ‘love’).
• Promoters — score 9–10/10 (dominant emotions are ‘delight’ and ‘excitement’).

This means that if the goal is to:

• reduce the number of detractors, you need to reduce their frustration and anger — doing this will turn them into passives;
• increase the number of promoters, you need to increase the excitement and delight among passives — doing this will turn them into promoters.

Figure 3: Full spectrum distribution of emotions across a typical consumer dataset

The acknowledged areas to target improvement or innovation exist at the extremes of the normal distribution curve,¹⁷ ie the deviance from the mean.¹⁸ In this example (Figure 3), that means innovation insights can be found at the extremes of positive and negative emotions.

Case study: Analysis of reviews of the Apple MacBook Pro

In the context of qualitative narrative data, my individual experience of the Apple MacBook Pro would be classified as anecdotal. However, if lots of people have similar experiences with the ‘ports’ then it becomes an evidence base that should not be ignored.

Pansensic (legally) scraped¹⁹ 1,236 online consumer reviews of real experiences of the Apple MacBook Pro and analysed the comments with the Pansensic Hybrid Text Analytics (HTA) platform. The first step of the analysis was to review the results of sentiment analysis over the main attributes of the products. As mentioned, this is the typical method employed by text analytics vendors. This process revealed that ‘ports’ were the eighth most talked about attribute, with 33 per cent negative sentiment (Figure 4). This is a useful and indicative high-level metric but should not be confused with insight.

Figure 4: Sentiment analysis across Apple MacBook Pro attributes from online consumer reviews

The next step was to analyse the same dataset using emotion analytics (Figure 5). Filtering against the main negative emotions: desire, frustration, anger, fear, sadness and shock, ‘ports’ rises to the second most talked about attribute.

Figure 5: Emotion analytics across Apple MacBook Pro from online consumer reviews, with negative emotion filtering

Drilling down into the comments associated with the negative emotions and ‘ports’ revealed a multitude of comments saying the same thing:

‘… only flaws are no USB cable or headphone jack’

‘… only problem is the USB port you have to buy an adapter and no CD drive, that was a deal breaker for me’

‘… the lack of USB ports and an SD card reader are annoying’

‘… my only complaint is the lack of ports and the price’

‘… i give it a 4 and not 5 star is the lack of USB ports that are still widely used’

‘… only thunderbolt 3 ports no USB’

‘… only real complaint is the lack of USB slots. there are two USB-C slots’

‘… my only complaint is that the there’s only thunderbolt ports you have to buy extra adapters not looking forward to having to always carry those around’

This insight regarding the perceived lack of ports was then reviewed against the aforementioned criteria, to see if it was actionable:

• relevant — yes;
• advantage — yes;
• trustworthy — yes;
• accurate — yes;
• holistic — no;
• granular — yes;
• novel — no.

It would be easy to conclude that Apple needs to address the lack of ports, but there is a bigger holistic picture to consider. Maybe the increase in revenue achieved by forcing consumers to buy additional, higher priced, adapters is of greater value to the organisation than losing a percentage of its customer base. Certainly, it is hard to imagine that Apple is unaware of any of the above experiences regarding ports on the MacBook Pro. This is where the ‘human in the loop’ steps in to make an informed decision with all available insights, both quantitative and qualitative, at their disposal.

At the same time, however, what a coup this provides for any competing laptop supplier choosing to analyse the online reviews of the Apple MacBook Pro. The commercial value of this analysis lies in identifying where it is possible to poach Apple passives and detractors by eliminating their frustrations by meeting their needs. Emotion analytics across the Apple MacBook Pro identifies that:

• promoters show 2 per cent frustration;
• passives show 16 per cent frustration; and
• detractors show 38 per cent frustration.

This should be seen as an evidence-based opportunity — look at all the causes of frustration and then design solutions and accordant marketing to inform the consumer that their needs have been met. As a particular example, in this data set, the second biggest frustration is lack of ports. HP lured me away from Apple by providing ten ports. Looking at my laptop right now as I type, five of them are in constant use. However, perhaps missing an opportunity to increase sales, HP did not advertise the number of ports on offer. I had to go looking for that solution.

HP, are you listening to Apple customers?

Conclusion

For the industry to drag itself out of the trough of disillusionment onto the slope of enlightenment and deliver organisational value it must go back to basics and be innovative in solving the fundamental text analytics problem that computers do not understand the meaning and context of words. To accept that, one way or another, the machine must learn meaning and context before it can begin to deliver commercial value on the plateau of productivity.

Carry on using sentiment analysis as a high-level metric tool as it is highly sensitive. However, emotion analytics is the best tool to extract improvement and innovation value from unstructured text. Emotion insights are more accurate and granular and thus considerably more actionable. Using emotion analytics to interrogate unstructured text delivers the greater commercial advantage.

The client needs to understand the difference between an insight and interesting information. Avoid getting caught up in the hype and seek commercial value by understanding exactly what makes an insight actionable.

Once actionable emotion insights have been obtained, the challenge remains to know which insight(s) to action. This requires input and decisions from a domain expert with intimate understanding of the business need to be solved. Any final decision should be the considered sum of a multi-perspective view of available, reliable and relevant input from enterprise and/or market-wide sources, and made by a human.

The Author

Paul Howarth is a world leader in extracting value from unstructured narrative datasets. In 2015, he founded Pansensic and led development of the Hybrid Text Analytics platform that leverages artificial intelligence technologies by teaching the machine with human curated ontologies, keyword sets and conditionalities. His experience is as an innovation, improvement and insight thought leader, having helped major global brands across multiple industries and sectors capitalise on insight extracted from their narrative datasets.

References

1. Adroit Market Research (2019) ‘Text Analytics Market will grow at 27.4% CAGR to reach $21.7 Billion by 2025 – Global Market Size and Share Analysis: Adroit Market Research’, available at: ://www.globenewswire.com/news-release/2019/04/16/1804648/0/en/Text-Analytics-Market-will-grow-at-27-4-CAGR-to-reach-21-7-Billion-by-2025-Global-Market-Size-and-Share-Analysis-Adroit-Market-Research.html (accessed 3rd January, 2020).
2. Prigg, M. (2019) ‘China’s leading AI expert warns the technology will take over HALF of jobs within 15 years’ available at: https://www.dailymail.co.uk/sciencetech/article-6575195/Chinas-leading-AI-expert-warns-HALF-jobs-15-years.html (accessed 3^rd January, 2020).
3. Woodie, A. (2018) ‘Mining New Opportunities in Text Analytics’, available at: https://www.datanami.com/2018/10/09/mining-new-opportunities-in-text-analytics/ (accessed 17^th February, 2020).
4. Goasduff, L. (2019) ‘Top Trends on the Gartner Hype Cycle for Artificial Intelligence, 2019’ available at: //www.gartner.com/smarterwithgartner/top-trends-on-the-gartner-hype-cycle-for-artificial-intelligence-2019/ (accessed 3^rd January, 2020).
5. Kim, S. (2018) ‘Highlights from the 2018 Google PhD Fellowship Summit’ available at: /ai.googleblog.com/2018/11 (accessed 3^rd January, 2020).
6. Lev-Ram, M. (2018) ‘’Why Thousands of Human Moderators Won’t Fix Toxic Content on Social Media’ available at: https://fortucom/2018/03/22/human-moderators-facebook-youtube-twitter/# (accessed 3^rd January, 2020).
7. Harris, Z. (1954) ‘Distributional Structure’, Word, Vol. 10, No. 2/3, pp. 146–62
8.  Mann, D.L. (2012) ‘Hearing What Is Really Being Said: Part 1 – Thinking Styles’ available at: http://www.systematic-innovation.com/assets/iss-127-oct-12.pdf (accessed 17^th February, 2020).
9. Mann, D.L. (2013) ‘Hearing What Is Really Being Said: Part 2 – Mercury’ available at: http://www.systematic-innovation.com/assets/iss-139-oct-13.pdf (accessed 17^th February, 2020).

10. Dykes, B. (2016) ‘Actionable Insights: The Missing Link Between Data And Business Value’ available at: https://www.forbes.com/sites/brentdykes/2016/04/26/actionable-insights-the-missing-link-between-data-and-business-value/#b73c2d651e57 (accessed 3^rd January, 2020).
11. Cunningham, J. (2017) ‘The Wisest Philosopher in Business: Peter Drucker’ available at: https://thunderbird.asu.edu/Knowledge-network/the-wisest-philosopher-in-business (accessed 17^th February, 2020).
12. Mäntylä, M., Graziotin, D. and Kuutila, M. ‘Evolution of Sentiment Analysis — A Review of Research Topics, Venues, and Top Cited Papers’ available at: https://www.researchgate.net/publication/311458740_The_Evolution_of_Sentiment_Analysis_-_A_Review_of_Research_Topics_Venues_and_Top_Cited_Papers (accessed17^th February, 2020).
13. BBC (2017) ‘Emojis honoured in world celebration’ available at: https://www.bbc.co.uk/news/technology-40612529 (accessed 17^th February, 2020).
14. Newman, D. (2015) ‘Customer Experience Is The Future Of Marketing’ available at: https://www.forbes.com/sites/danielnewman/2015/10/13/customer-experience-is-the-future-of-marketing/ (accessed 17^th February, 2020).
15. Haque, A. (2004) ‘Psychology from Islamic perspective: contributions of early Muslim Scholars and challenges to contemporary Muslim psychologists’, Journal of Religion and Health, Vol. 43, No. 4, pp. 357–377.
16. Gaulin, S.J.C. and McBurney, D.H. (2003) ‘Evolutionary Psychology’, Prentice Hall, Lebanon, IN, pp. 121–142.
17. Mann, D.L. (2007) ‘Hands on systematic innovation’, 2nd edn, IFR Press, Clevedon, England.
18. Mathews, R. and Wacker, W. (2003) ‘The Deviants Advantage: How Fringe Ideas Create Mass Markets’, Random House Business Books, New York, NY.
19. United States Court Of Appeals For The Ninth Circuit (2019) ‘Preliminary Injunction / Computer Fraud and Abuse Act’ https://cdn.ca9.uscourts.gov/datastore/opinions/2019/09/09/17-16783.pdf (accessed17^th February, 2020).