Explained: XLNet – Generalized Autoregressive Pretraining for Language Understanding

Carnegie Mellon and Google’s Brain outfit have tried to undo some of the techniques of Google’s BERT machine learning model for natural language processing.

They propose a new approach called “XLNet.” Built on top of the popular “Transformer” A.I. for language, it may be a more straightforward way to examine how language works.

XLNet is an exciting development in NLP, not only because of its results but because it shows us that there is still room to improve upon for transfer learning in NLP.

Machine Learning Explained’s article, Paper Dissected: “XLNet: Generalized Autoregressive Pretraining for Language Understanding” Explained, offers a clear summary of arguably one of 2019’s most important developments in Natural Language Processing.

Further Readings

The original paper

Blog post on the Transformer

Blog post on ELMo

Blog post on BERT

Bots for Humanity

99 percent of technological progress by modern humans came in the last 10,000 years. We took tens of thousands of years to settle down, after a period of global migration. Once we did, we discovered ways to cultivate plants, around 12,000 years ago, discovered metals, around 8,000 years ago, and started writing things, around 5,000 years ago. Each of these phases helped us bring mankind together through teaching, with the last phase being one that allowed us to pass ideas on beyond the lifetime of the teacher.

It is in this passing of information through generations of teaching that has lead us to arrive at the technologically advanced times that we live in today but ever growing risks from existential threats are pacing towards us at a rate much faster than that of the progress of our countering efforts. Finding better and faster solutions to some of mankind’s most pressing issues may be beyond the capacity of collective human minds. In the wake of an Artificial Intelligence explosion, could mankind be saved by heroic bots, created to discover the solutions we can’t?

Last month, a neuroscience-inspired, Artificial Intelligence company called Circle AI, collaborated with WPP to create a conversation bot for the United Nations. The aim of the bot was to get citizens to track the daily progress of 10 actions that were identified as positive acts we can all take to help tackle climate change.

The initiative was first put forward by Michael Møller, Director-General of the United Nations Office at Geneva (UNOG). He said: “The challenge to humanity that climate change represents is of such epic proportions that only through collective global action will we have a chance to combat it successfully. Every single human being on our severely stressed planet has to take responsibility.”

Circle AI Chief Executive and AI Research Engineer, Jason Hadjioannou said: “Mitigating the devastatingly catastrophic effects of climate change and tackling other existential threats to Humans and planet Earth, may take the someday invention of Artificial General Intelligence. But today we can make a start by creating Artificial Intelligence systems and AI agents that help.”

The United Nations’ conversation bot, named the ActNowBot, was announced by Sir David Attenborough at the United Nations Climate Change Conference in Poland on the 3rd December 2018.

Speaking about the project, Sir David Attenborough said: “We all know climate change is a global problem – and that it requires a global solution. This is an opportunity for people from across the globe, regardless of their nationality or circumstances, to be part of most important discussion of this century: the unprecedented action needed to reach the Paris Agreement targets.”

London-based, Circle AI provides conversational AI tech to organisations like Facebook Inc and the UN. Circle also concerns itself with contributing R&D efforts towards the exploration of ideas around Artificial General Intelligence (AGI), crediting some of the inspiration for concepts on conversational AI to the ideas and work of British-American philosopher and cognitive scientist, Dr Peter Carruthers and his theories in human cognition – specifically quoting: “there is a type of inner, explicitly linguistic thinking that allows us to bring our own thoughts into conscious awareness. We may be able to think without language, but language lets us know that we are thinking.”

Hadjioannou cites motivation for wanting to contribute towards advancing the field of AGI, as coming from the notion that solutions to some of mankind’s greatest problems may be beyond that of Human minds, instead someday emerging from the limitless ingenuity and advanced cognitive ability of artificial minds.

With Apple’s CEO, Tim Cook recently commenting that Apple’s “most important contribution to mankind will be in health” it seems there is a burgeoning trend to look deep into the future of cutting-edge technology for ways to save humanity. And this is reassuringly exciting!

Stanford Introduces it’s Human-Centered AI Initiative

A common goal for the brightest minds from Stanford and beyond: putting humanity at the center of AI.

— Fei-Fei Li & John Etchemendy [http://hai.stanford.edu]

Humanity: The Next Frontier in AI

We have arrived at a truly historic turning point: Society is being reshaped by technology faster and more profoundly than ever before. Many are calling it the fourth industrial revolution, driven by technologies ranging from 5G wireless to 3D printing to the Internet of Things. But increasingly, the most disruptive changes can be traced to the emergence of Artificial Intelligence.

Many of these changes are inspiring. Machine translation is making it easier for ideas to cross language barriers; computer vision is making medical diagnoses more accurate; and driver-assist features have made cars safer. Other changes are more worrisome: Millions face job insecurity as automation rapidly evolves; AI-generated content makes it increasingly difficult to tell fact from fiction; and recent examples of bias in machine learning have shown us how easily our technology can amplify prejudice and inequality.

Like any powerful tool, AI promises risk and reward in equal measure. But unlike most “dual-use” technologies, such as nuclear energy and biotech, the development and use of AI is a decentralized, global phenomenon with a relatively low barrier to entry. We can’t control something so diffuse, but there is much we can do to guide it responsibly. This is why the next frontier in AI cannot simply be technological—it must be humanistic as well.

The Stanford Human-Centered AI Initiative (HAI)

Many causes warrant our concern, from climate change to poverty, but there is something especially salient about AI: Although the full scope of its impact is a matter of uncertainty, it remains well within our collective power to shape it. That’s why Stanford University is announcing a major new initiative to create an institute dedicated to guiding the future of AI. It will support the necessary breadth of research across disciplines; foster a global dialogue among academia, industry, government, and civil society; and encourage responsible leadership in all sectors. We call this perspective Human-Centered AI, and it flows from three simple but powerful ideas:

  1. For AI to better serve our needs, it must incorporate more of the versatility, nuance, and depth of the human intellect.
  2. The development of AI should be paired with an ongoing study of its impact on human society, and guided accordingly.
  3. The ultimate purpose of AI should be to enhance our humanity, not diminish or replace it.

Realizing these goals will be among the greatest challenges of our time. Each presents complex technical challenges and will provoke dialogues among engineers, social scientists, and humanists. But this raises some important questions: What are the most pressing problems, who will solve these problems, and where will these dialogues take place?

Human-Centered AI requires a broad, multidisciplinary effort that taps the expertise of an extraordinary range of disciplines, from neuroscience to ethics. Meeting this challenge will require us to take chances exploring uncertain new terrain with no promise of a commercial product. It is far more than an engineering task.

The Essential Role of Academia

This is the domain of pure research. It’s the scientific freedom that allowed hundreds of universities to collaborate internationally to build the Large Hadron Collider—not to make our phones cheaper or our Wi-Fi faster, but to catch the first glimpse of the Higgs boson. It’s how we built the Hubble Telescope and mapped the human genome. Best of all, it’s inclusive; rather than compete for market share, it invites us to work together for the benefit of deeper understanding and knowledge that can be shared.

Even more important, academia is charged with educating the leaders and practitioners of tomorrow across a range of disciplines. The evolution of AI will be a multigenerational journey, and now is the time to instill human-centered values in the technologists, engineers, entrepreneurs, and policy makers who will chart its course in the years to come.

Why Stanford?

Realizing the goals of Human-Centered AI will require cooperation between academia, industry, and governments around the world. No single university will provide all the answers; no single company will define the standards; no single nation will control the technology.

Still, there is a need for a focal point, a center specifically devoted to the principles of Human-Centered AI, capable of rapidly advancing the research frontier and acting as a global clearinghouse for ideas from other universities, industries, and governments. We believe that Stanford is uniquely suited to play this role.

Stanford has been at the forefront of AI since John McCarthy founded the Stanford AI Lab (SAIL) in 1963. McCarthy coined the term “Artificial Intelligence” and set the agenda for much of the early work in the field. In the decades since, SAIL has served as the backdrop for many of AI’s greatest milestones, from pioneering work in expert systems to the first driverless car to navigate the 130-mile DARPA Grand Challenge. SAIL was the home of seminal work in computer vision and the birthplace of ImageNet, which demonstrated the transformative power of large-scale datasets on neural network algorithms. This tradition continues today, with active research by more than 100 doctoral students, as well as many master’s students and undergraduates. Research topics include computer vision, natural language processing, advanced robotics, and computational genomics.

But guiding the future of AI requires expertise far beyond engineering. In fact, the development of Human-Centered AI will draw on nearly every intellectual domain—and this is precisely what makes Stanford the ideal environment to enable it. The Stanford Law School, consistently regarded as one of the world’s most prestigious, brings top legal minds to the debate about the future of ethics and regulation in AI. Stanford’s social science and humanities departments, also among the strongest in the world, bring an understanding of the economic, sociological, political, and ethical implications of AI. Stanford’s Schools of Medicine, Education, and Business will help explore how intelligent machines can best serve the needs of patients, students, and industry. Stanford’s rich tradition of leadership across the disciplinary spectrum will allow us to chart the future of AI around human needs and interests.

Finally, Stanford’s location—both in the heart of Silicon Valley and on the Pacific Rim—places it in close proximity to many of the companies leading the commercial revolution in AI. With deeper roots in Silicon Valley than any other institution, Stanford can both learn from and share insights with the companies most capable of influencing that revolution.

With the Human-Centered AI Initiative, Stanford aspires to become home to a vibrant coalition of thinkers working together to make a greater impact than would be possible on their own. This effort will be organized around five interrelated goals:

  • Catalyze breakthrough, multidisciplinary research.
  • Foster a robust, global ecosystem.
  • Educate and train AI leaders in academia, industry, government, and civil society.
  • Promote real-world actions and policies.
  • And, perhaps most important, stimulate a global dialogue on Human-Centered AI.

In Closing

For decades AI was an academic niche. Then, over just a few years, it emerged as a powerful tool capable of reshaping entire industries. Now the time has come to transform it into something even greater: a force for good. With the right guidance, intelligent machines can bring life-saving diagnostics to the developing world, provide new educational opportunities in underserved communities, and even help us keep a more vigilant eye on the health of the environment. The Stanford Human-Centered AI Initiative is a large-scale effort to make these visions, and many more, a reality. We hope you’ll join us.

Visit http://hai.stanford.edu to find out more

Overcoming Priors for Visual Question Answering

Abstract

A number of studies have found that today’s Visual Question Answering (VQA) models are heavily driven by superficial correlations in the training data and lack sufficient image grounding. To encourage development of models geared towards the latter, we propose a new setting for VQA where for every question type, train and test sets have different prior distributions of answers. Specifically, we present new splits of the VQA v1 and VQA v2 datasets, which we call Visual Question Answering under Changing Priors (VQA-CP v1 and VQA-CP v2 respectively). First, we evaluate several existing VQA models under this new setting and show that their performance degrades significantly compared to the original VQA setting. Second, we propose a novel Grounded Visual Question Answering model (GVQA) that contains inductive biases and restrictions in the architecture specifically designed to prevent the model from ‘cheating’ by primarily relying on priors in the training data. Specifically, GVQA explicitly disentangles the recognition of visual concepts present in the image from the identification of plausible answer space for a given question, enabling the model to more robustly generalize across different distributions of answers. GVQA is built off an existing VQA model – Stacked Attention Networks (SAN). Our experiments demonstrate that GVQA significantly outperforms SAN on both VQA-CP v1 and VQA-CP v2 datasets. Interestingly, it also outperforms more powerful VQA models such as Multimodal Compact Bilinear Pooling (MCB) in several cases. GVQA offers strengths complementary to SAN when trained and evaluated on the original VQA v1 and VQA v2 datasets. Finally, GVQA is more transparent and interpretable than existing VQA models.

Aishwarya AgrawalDhruv BatraDevi Parikh, Aniruddha Kembhavi
[Facebook AI Research]

Download the full paper here

The importance of single directions for generalization

ABSTRACT

Despite their ability to memorize large datasets, deep neural networks often achieve good generalization performance. However, the differences between the learned solutions of networks which generalize and those which do not remain unclear. Additionally, the tuning properties of single directions (defined as the activation of a single unit or some linear combination of units in response to some input) have been highlighted, but their importance has not been evaluated. Here, we connect these lines of inquiry to demonstrate that a network’s reliance on single directions is a good predictor of its generalization performance, across networks trained on datasets with different fractions of corrupted labels, across ensembles of networks trained on datasets with unmodified labels, across different hyperparameters, and over the course of training. While dropout only regularizes this quantity up to a point, batch normalization implicitly discourages single direction reliance, in part by decreasing the class selectivity of individual units. Finally, we find that class selectivity is a poor predictor of task importance, suggesting not only that networks which generalize well minimize their dependence on individual units by reducing their selectivity, but also that individually selective units may not be necessary for strong network performance.

Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, & Matthew Botvinick
@DeepMind

Download the full paper here

SingularityNET: A decentralized, open market and inter-network for AIs

ABSTRACT

The value and power of Artificial Intelligence is growing dramatically every year, and will soon dominate the internet – and the economy as a whole. However, AI tools today are fragmented by a closed development environment; most are developed by one company to perform one task, and there is no way to plug two tools together. SingularityNET aims to become the key protocol for networking AI and machine learning tools to form a coordinated Artificial General Intelligence. SingularityNET is an open-source protocol and collection of smart contracts for a decentralized market of coordinated AI services. Within this framework, the benefits of AI become a global commons infrastructure for the benefit of all; anyone can access AI tech or become a stakeholder in its development. Anyone can add an AI/machine learning service to SingularityNET for use by the network, and receive network payment tokens in exchange. SingularityNET is backed by the SingularityNET Foundation, which operates on a belief that the benefits of AI should not be dominated by any small set of powerful institutions, but shared by all. A key goal of SingularityNET is to ensure the technology is benevolent according to human standards, and the network is designed to incentivize and reward beneficial players.

Ben Goertzel

Download the full paper here

WaveNet: A Generative Model for Raw Audio

ABSTRACT

This paper introduces WaveNet, a deep neural network for generating raw audio waveforms. The model is fully probabilistic and autoregressive, with the predictive distribution for each audio sample conditioned on all previous ones; nonetheless we show that it can be efficiently trained on data with tens of thousands of samples per second of audio. When applied to text-to-speech, it yields state-ofthe-art performance, with human listeners rating it as significantly more natural sounding than the best parametric and concatenative systems for both English and Mandarin. A single WaveNet can capture the characteristics of many different speakers with equal fidelity, and can switch between them by conditioning on the speaker identity. When trained to model music, we find that it generates novel and often highly realistic musical fragments. We also show that it can be employed as a discriminative model, returning promising results for phoneme recognition.

Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukcuoglu
{avdnoord, sedielem, heigazen, simonyan, vinyals, gravesa, nalk, andrewsenior, korayk}@google.com
Google DeepMind

Download the full paper here

AI XPRIZE – AI competition with IBM Watson

The IBM Watson AI XPRIZE is a $5 million AI and cognitive computing competition challenging teams globally to develop and demonstrate how humans can collaborate with powerful AI technologies to tackle the world’s grand challenges. This prize will focus on creating advanced and scalable applications that benefit consumers and businesses across a multitude of disciplines. The solutions will contribute to the enrichment of available tools and data sets for the usage of innovators everywhere. The goal is also to accelerate the understanding and adoption of AI’s most promising breakthroughs.

Every year leading up to TED 2020, the teams will compete for interim prizes and the opportunity to advance to the next year’s competition. The three finalist teams will take the TED stage in 2020 to deliver jaw-dropping, awe-inspiring TED Talks demonstrating what they have achieved.

Typical of all XPRIZE competitions, the IBM Watson AI XPRIZE will crowdsource solutions from some of the most brilliant thinkers and entrepreneurs around the world, creating true exponential impact.

To compete in the IBM Watson AI XPRIZE you must be a fully registered team. To complete your registration, you must create a Team profile, sign the Competitor’s Agreement and pay the registration fee.

AI Xprize Timeline

PRIZE PURSE

Grand Prizes

The $3,000,000 Grand Prize, $1,000,000 2nd Place, and $500,000 3rd Place purses will be awarded at the end of competition at TED2020, for a total of $4.5 million.

Milestone and Special Prizes

Two Milestone Competition prize purses will be awarded at the end of each of the first two rounds of the competition, and the Judges may award additional special prizes to recognize special accomplishments. A total of $500,000 will be available for these prizes and will be allocated by the Judges for special accomplishments.

THE NEED FOR THE PRIZE

The progress in AI research and applications in the past 20 years makes it timely to focus attention not only on making AI more capable, but also on maximizing the societal benefit of AI. The democratization of exponential technology enables AI and cognitive computing to put empowerment into the hands of innovators everywhere. Driven by long term capabilities of AI impact, and to better understand the prospects of human and AI collaboration, the IBM Watson AI XPRIZE provides an interdisciplinary platform for domain experts, developers and innovators to, through collaboration, push the boundaries of AI to new heights. The competition will bring the AI community together and accelerate the development of scalable, hybrid solutions and audacious breakthroughs to address humanity’s grandest challenges.

You can register for the competition at: https://aiportal.xprize.org/en/registration

Watch AlphaGo take on Lee Sedol, the world’s top Go player

Watch AlphaGo take on Lee Sedol, the world’s top Go player, in the final match of the Google DeepMind challenge.

Match score: AlphaGo 3 – Lee Sedol 1.
[Game five: Seoul, South Korea, 15th March at 13:00 KST; 04:00 GMT; for US at -1 day (14th March) 21:00 PT, 00:00 ET.]

The Game of Go 

The game of Go originated in China more than 2,500 years ago. The rules of the game are simple: Players take turns to place black or white stones on a board, trying to capture the opponent’s stones or surround empty space to make points of territory. As simple as the rules are, Go is a game of profound complexity. There are more possible positions in Go than there are atoms in the universe. That makes Go a googol times more complex than chess. Go is played primarily through intuition and feel, and because of its beauty, subtlety and intellectual depth it has captured the human imagination for centuries. AlphaGo is the first computer program to ever beat a professional, human player. Read more about the game of Go and how AlphaGo is using machine learning to master this ancient game.

Match Details 

In October 2015, the program AlphaGo won 5-0 in a formal match against the reigning 3-times European Champion, Fan Hui, to become the first program to ever beat a professional Go player in an even game. Now AlphaGo will face its ultimate challenge: a 5-game challenge match in Seoul against the legendary Lee Sedol, the top Go player in the world over the past decade, for a $1M prize. For full details, see the press release.

The matches were held at the Four Seasons Hotel, Seoul, South Korea, starting at 13:00 local time (04:00 GMT; day before 20:00 PT, 23:00 ET) on March 9th, 10th, 12th, 13th and 15th.

The matches were livestreamed on DeepMind’s YouTube channel as well as broadcast on TV throughout Asia through Korea’s Baduk TV, as well as in China, Japan, and elsewhere.Match commentators included Michael Redmond, the only professional Western Go player to achieve 9 dan status. Redmond commentated in English, and Yoo Changhyuk professional 9 dan, Kim Sungryong professional 9 dan, Song Taegon professional 9 dan, and Lee Hyunwook professional 8 dan commentated in Korean alternately.The matches were played under Chinese rules with a komi of 7.5 (the compensation points the player who goes second receives at the end of the match). Each player received two hours per match with three lots of 60-second byoyomi (countdown periods after they have finished their allotted time).

Singularity Or Bust [Documentary]

In 2009, film-maker and former AI programmer Raj Dye spent his summer following futurist AI researchers Ben Goertzel and Hugo DeGaris around Hong Kong and Xiamen, documenting their doings and gathering their perspectives. The result, after some work by crack film editor Alex MacKenzie, was the 45 minute documentary Singularity or Bust — a uniquely edgy, experimental Singularitarian road movie, featuring perhaps the most philosophical three-foot-tall humanoid robot ever, a glance at the fast-growing Chinese research scene in the late aughts, and even a bit of a real-life love story. The film was screened in theaters around the world, and won the Best Documentary award at the 2013 LA Cinema Festival of Hollywood and the LA Lift Off Festival. And now it is online, free of charge, for your delectation.

Singularity or Bust is a true story pertaining to events occurring in the year 2009. It captures a fascinating slice of reality, but bear in mind that things move fast these days. For more recent updates on Goertzel and DeGaris’s quest for transhuman AI, you’ll have to consult the Internet, or your imagination.

[Full Documentary]