Facebook Using AI to Describe Photos to the Blind

As any user of Facebook will immediately tell you, photos play a major function in sharing of life stories. From hiking mountains to capturing the beauty of a well-prepared dinner, photos add to the story tremendously. After all, a picture is worth a thousand words, right? (And thus the success of emoticons, I think).

But what if those photos cannot be seen by visually challenged FB friends? Not to worry. Facebook’s engineers have harnessed the power of an artificial intelligence network (AI)  to describe these pictures to blind or partially blind users.

Facebook calls the system “automatic alternative text” and it’s based on a neural network primed with billions of parameters and millions of examples. Such neural networks – vast, complex databases designed to mimic the human brain as closely as possible – are playing an increasingly important role in modern computing.

The AI doesn’t actually “see” what is in the photo, and, as with all things computer at the moment, doesn’t understand context (and context is vital for a true understanding of photos). Rather, the AI compares the objects in the image with its “its vast internal database of similar photos and make an educated guess about what’s being shown.”

As to context, part of the challenge is in getting computers to recognize what’s most important in an image, whether that’s the people, the background, or the “action.” This requires a great more programming, but that’s in the future.

For now, the AI system returns a confidence score indicating how sure it is that it can identify what’s in the picture. If this is above 80 percent, an automatically-generated caption appears. 


When objects and people have been identified, Facebook’s software constructs a sentence to describe the picture, usually ordered by how confident the AI is about the presence of each element. If there’s some ambiguity about the picture then the sentence starts with “image may contain” to express that uncertainty.

The feature is live now in the Facebook iOS app, as long as your language is set to English and you’re in the US, UK, Canada, Australia, or New Zealand. Facebook says it hopes to roll out the service to more platforms, languages and markets in the near future. It actually works with any screen reader software – on iOS you can enable it via the VoiceOver tool in the Accessibility section of Settings (under General), for example.

I wonder what the AI writes about such photos:



Call a Doc

So, you have a sore throat but you can’t get in to see a doctor for a few days. What do you do?

If things advance as technology allows, you can phone your doctor or contact her on the Internet, and be attended to virtually. It’s called “telemedicine” and the idea is gaining traction around the world. According to an article in the Wall Street Journal:

“Driven by faster internet connections, ubiquitous smartphones and changing insurance standards, more health providers are turning to electronic communications to do their jobs—and it’s upending the delivery of health care.”

In some cases, it makes sense.

“Doctors are linking up with patients by phone, email and webcam. They’re also consulting with each other electronically—sometimes to make split-second decisions on heart attacks and strokes. Patients, meanwhile, are using new devices to relay their blood pressure, heart rate and other vital signs to their doctors so they can manage chronic conditions at home.”

It’s not just locally that this makes sense, either.

“Telemedicine also allows for better care in places where medical expertise is hard to come by. Five to 10 times a day, Doctors Without Borders relays questions about tough cases from its physicians in Niger, South Sudan and elsewhere to its network of 280 experts around the world, and back again via the internet.”

While it might seem a new idea, “More than 15 million Americans received some kind of medical care remotely last year, according to the American Telemedicine Association, a trade group, which expects those numbers to grow by 30% this year.”

But not all is bright on the horizon. There are serious questions about the quality and qualifications of the doctors who practice telemedicine, and questions about who will foot the cost. “Some critics … question whether the quality of care is keeping up with the rapid expansion of telemedicine. And there’s the question of what services physicians should be paid for: Insurance coverage varies from health plan to health plan, and a big federal plans covers only a narrow range of services.”

“Critics worry that such services may be sacrificing quality for convenience. Consulting a random doctor patients will never meet, they say, further fragments the health-care system, and even minor issues such as upper respiratory infections can’t be thoroughly evaluated by a doctor who can’t listen to your heart, culture your throat or feel your swollen glands.”

As to the question of qualifications, the WSJ discusses one example:

“First Opinion, connects users with doctors in India for web chats, but a disclaimer states that these are merely ‘social interactions.’ If a prescription or lab test is warranted, a locally licensed doctor joins the conversation for a $39 fee. The company didn’t respond to requests for comment.

“Are such services ‘practicing medicine’ without a license? The exact definition varies from state to state, and state medical boards generally don’t investigate unless a patient files a formal complaint. Even then, boards have jurisdiction only over individual doctors licensed in their state, not companies, or physicians overseas, says Ms. Robbin of the Federation of State Medical Boards.”

There’s also the problem of “branding” medical providers (though we have that already, with mega-medical groups). “Telemedicine is also shaking up traditional relationships between providers and payers and fueling the rise of medical ‘megabrands’ whose experts are increasingly competing for patients in each other’s backyards.”

It will be interesting to see where all of this leads. People are not happy with the current state of insurance and medical care available in the US, and elsewhere in the world. What Brave New World awaits us on this front?