Help Shape our AI Summaries: Participate in Survey #1

Survey #1: Select the Best Summary and Impact Our Research

Hi everyone! I would like to invite you to a survey. You will be given a piece of original text and 4 options to choose the best summary. You can only choose one option to be able to vote.

The survey outcome will help me tune hyperparameters, determine which algorithms performed the best in which scenarios, and determine whether users prefer concise or detailed summaries, among other things.

Thank you for your time, and I look forward to see the results.


Choose the best summary from the article sources

Example 1 - Tiger

Source: Tiger - Wikipedia

Summaries - Click to expand
  • Summary 1
    The tiger is the largest living felid species, with the Siberian and Bengal tigers being the largest.[47] Bengal tigers average a total length of 3 m (9.8 ft), with males weighing 200–260 kg (440–570 lb) and females weighing 100–160 kg (220–350 lb).
    Island tigers are the smallest; the Sumatran tiger has a total length of 2.2–2.5 m (7 ft 3 in – 8 ft 2 in) with a weight of 100–140 kg (220–310 lb) for males and 75–110 kg (165–243 lb) for females.[51] The extinct Bali tiger was even smaller.[47] It has been hypothesised that body sizes of different tiger populations may be correlated with climate and be explained by thermoregulation and Bergmann's rule.[47][11]
  • Summary 2
    The tiger (Panthera tigris) is the largest living cat species and a member of the genus Panthera native to Asia. It has a powerful, muscular body with a large head and paws, a long tail, and orange fur with black, mostly vertical stripes. It is traditionally classified into nine recent subspecies, though some recognise only two subspecies, mainland Asian tigers and island tigers of the Sunda Islands.
    The tiger has a typical felid morphology; its body is muscular with shortened legs, strong forelimbs, broad paws, a large head and a tail that is about half the length of the rest of its body.[47][48] There are five digits on the front feet and four on the back, all of which have retractile claws that are compact and curved.
    The tiger is the largest living felid species, with the Siberian and Bengal tigers being the largest.[47] Bengal tigers average a total length of 3 m (9.8 ft), with males weighing 200–260 kg (440–570 lb) and females weighing 100–160 kg (220–350 lb). Island tigers are the smallest; the Sumatran tiger has a total length of 2.2–2.5 m (7 ft 3 in – 8 ft 2 in) with a weight of 100–140 kg (220–310 lb) for males and 75–110 kg (165–243 lb) for females.[51] The extinct Bali tiger was even smaller.[47] It has been hypothesised that body sizes of different tiger populations may be correlated with climate and be explained by thermoregulation and Bergmann's rule.[47][11]
  • Summary 3
    The tiger featured prominently in the ancient mythology and folklore of cultures throughout its historic range and has continued to appear in culture worldwide.
    The tiger has a typical felid morphology; its body is muscular with shortened legs, strong forelimbs, broad paws, a large head and a tail that is about half the length of the rest of its body.[47][48] There are five digits on the front feet and four on the back, all of which have retractile claws that are compact and curved. The ears are rounded, while the eyes have a round pupil.[47] The tiger's skull is large and robust, with a constricted front region, proportionally small, elliptical orbits, long nasal bones, and a lengthened cranium with a large sagittal crest.[49][47] It resembles a lion's skull, with the structure of the lower jaw and length of the nasals being the most reliable indicators for species identification.[49].
    The tiger has fairly robust teeth and its somewhat curved canines are the longest in the cat family at 6.4–7.6 cm (2.5–3.0 in).[47][50] The tiger is the largest living felid species, with the Siberian and Bengal tigers being the largest.[47].
    Bengal tigers average a total length of 3 m (9.8 ft), with males weighing 200–260 kg (440–570 lb) and females weighing 100–160 kg (220–350 lb). Island tigers are the smallest; the Sumatran tiger has a total length of 2.2–2.5 m (7 ft 3 in – 8 ft 2 in) with a weight of 100–140 kg (220–310 lb) for males and 75–110 kg (165–243 lb) for females.[51] The extinct Bali tiger was even smaller.[47] It has been hypothesised that body sizes of different tiger populations may be correlated with climate and be explained by thermoregulation and Bergmann's rule.[47][11]
  • Summary 4
    Tigers are also victims of human–wildlife conflict for attacking and preying on livestock in areas, where natural prey is scarce. The tiger is among the most popular of the world's charismatic megafauna. The tiger (Panthera tigris) is the largest living cat species and a member of the genus Panthera native to Asia.
    When becoming independent, they leave their mother's home range and establish their own. Females give birth to usually two or three cubs that stay with their mother for about two years. The ears are rounded, while the eyes have a round pupil.[47].
    The tiger's skull is large and robust, with a constricted front region, proportionally small, elliptical orbits, long nasal bones, and a lengthened cranium with a large sagittal crest.[49][47] It resembles a lion's skull, with the structure of the lower jaw and length of the nasals being the most reliable indicators for species identification.[49] The tiger has fairly robust teeth and its somewhat curved canines are the longest in the cat family at 6.4–7.6 cm (2.5–3.0 in).[47][50].
    Island tigers are the smallest; the Sumatran tiger has a total length of 2.2–2.5 m (7 ft 3 in – 8 ft 2 in) with a weight of 100–140 kg (220–310 lb) for males and 75–110 kg (165–243 lb) for females.[51] The extinct Bali tiger was even smaller.[47] It has been hypothesised that body sizes of different tiger populations may be correlated with climate and be explained by thermoregulation and Bergmann's rule.[47][11] It is traditionally classified into nine recent subspecies, though some recognise only two subspecies, mainland Asian tigers and island tigers of the Sunda Islands.
0 voters

Example 2 - Pooping on the Moon is a messy business

Source: Pooping on the Moon Is a Messy Business | WIRED

Summaries - Click to expand
  • Summary 1
    It’s not what we want to do. Current rules under the Committee on Space Research, an international nongovernmental organization that promotes worldwide collaboration on scientific research in space, prohibit activities that could allow either form of contamination. NASA’s contest was a brainstorming effort, so it’s not clear at this time how many of the winner’s or finalists’ design features will appear in the ultimate Lunar Loo, or how Artemis crews will manage and dispose of waste on the surface.
    Meanwhile, growing numbers of livestock worldwide, and the billions of tons of feces they produce each year, are straining waste management programs. “The contents of these bags could, in the long run, cause major concerns for the lunar environment as well as for the human scientific activities carried out on the moon.”
    Former NASA Astronaut Answers Space Questions From Twitter “Basically, in space a human no longer has gravity to assist pulling the feces away from the anus,” explains David Munns, a professor of the history of science and technology at John Jay College, City University of New York, who recently coauthored a book about waste management in space with Kärin Nickelsen, a professor at Ludwig Maximilian University of Munich. “It would be another data point that says it’s a little bit easier to believe that life can exist in lots of places throughout the galaxy, solar system, and universe at large.”
  • Summary 2
    Former NASA Astronaut Answers Space Questions From Twitter The winning concept, entitled Translunar Hypercritical Repository 1, or THRONE, was inspired in part by the diaper genie product that will be familiar to many parents of toddlers, which seals waste in plastic as soon as it is disposed, to reduce smell. These concerns fall under the umbrella of planetary protection, a term that encompasses both “forward” contamination, in which Earth life is transferred to an extraterrestrial surface, and “backward” contamination, which refers to hypothetical alien life that is brought back to Earth.
    But despite the taboo nature of the topic, the team emphasized its centrality to human spaceflight, noting that “the astronauts have learned quickly the importance of gravity in the mechanics of defecation.”
    Human feces is packed with microbial life, which means that the moon hosted life on its surface for an unknown period of time after each Apollo landing. So the question is, what are the environmental standards that we will apply? But in micro- or lunar gravity, waste does not disconnect from the body so easily, and it can behave unpredictably in storage, inspiring memorable phrases such as “fecal popcorning,” referring to the movements of astro-poop as it bounces off the sides of space toilet containment tanks. “It becomes really a sticky liquid problem of surface tension.
  • Summary 3
    The bags also raise questions about our cultural heritage and environmental impact on the lunar environment, while underscoring the intractable problem of managing and disposing of off-Earth biological waste. These topics are more salient than ever now that there’s a renewed push, both from governments and commercial actors, to return humans to the surface of the moon, potentially for stays of weeks, months, or—in the most optimistic visions—indefinitely.
    Former NASA Astronaut Answers Space Questions From Twitter “If there’s going to be humans living permanently on the moon, you don’t want bags of poo lying around,” says Melissa de Zwart, a professor at the University of Adelaide who specializes in the legal and regulatory aspects of outer space or environments, including the moon.
    But scientists haven’t developed a circular system that can dispose of all the biological waste produced by humans in space—urine, feces, vomit, and menstrual blood—which is a major technology gap for future human space exploration. Now, NASA is spearheading the Artemis Program, an international effort to return humans to the lunar surface, and China’s space agency also aims to land crews on the moon in the coming decade. “One of the key challenges for NASA and its Artemis partners during these next generation lunar missions is developing a waste management system which can not only successfully collect waste in a weightlessness environment, like on the International Space Station in low Earth orbit, but also in one-sixth gravity while on the lunar surface.” Whatever Artemis’s final design, it will incorporate more than 50 years of innovation in spaceflight.
    Recently, the Native American Navajo Nation expressed its opposition to a private lunar lander taking cremated human remains to the moon, calling the practice a “profound desecration.” The mission, Astrobiotic’s Peregrine Mission One, failed to reach the lunar surface, but this conflict in values may flare up again as surface missions proliferate, and it could ultimately encompass biological waste on the moon. In addition to raising these legal and ethical quandaries, the Apollo waste bags have also inspired exciting scientific questions.
  • Summary 4
    The Apollo crews left a total of 96 bags of waste, including urine and feces, across their six landing sites, which are still sitting there to this day: a celestial reminder that wherever humans go, we bring our shit with us. These Apollo jettison bags, sometimes shorthanded as the “poo bags,” have been the subject of much interest and speculation since they were deposited on the moon more than 50 years ago. “If there’s going to be humans living permanently on the moon, you don’t want bags of poo lying around,” says Melissa de Zwart, a professor at the University of Adelaide who specializes in the legal and regulatory aspects of outer space or environments, including the moon.
    But where should this waste go, and what should be done with it? “The contents of these bags could, in the long run, cause major concerns for the lunar environment as well as for the human scientific activities carried out on the moon.” Analyzing these samples could reveal whether the microbes in the bags survived or even dispersed into the wider lunar environment, which is critical information for future exploration.
    These concerns fall under the umbrella of planetary protection, a term that encompasses both “forward” contamination, in which Earth life is transferred to an extraterrestrial surface, and “backward” contamination, which refers to hypothetical alien life that is brought back to Earth. “We bring with us nonhuman passengers, like microbes and bacteria, as well as our own bodies and the things that go in and out of them.
0 voters

Example 3 - Why Do India and China Keep Fighting Over This Desolate Terrain?

Source: https://www.nytimes.com/2024/06/27/magazine/india-china-border.html

Summaries - Click to expand
  • Summary 1
    “One is the Indian perception of the Line of Actual Control. Another is the Chinese perception of the Line of Actual Control. Third is the Indian perception of the Chinese perception of the Line of Actual Control — because we have a perception based on their line of patrolling. And the fourth is, of course, the Chinese perception of the Indian perception of the Line of Actual Control.” Please go back,’” Bhatia told me. “And they hold up a banner of their own that says in Hindi and in English: ‘You are in our territory.
    In recent years, however, confrontations have sometimes spiraled into skirmishes. “This conflict isn’t going to go away in a hurry,” he told me. Less than a year and a half later, days before Xi’s state visit to India in 2014, Chinese troops entered Chumar, in eastern Ladakh. China claims Doklam as its territory, while India and Bhutan maintain that the area is a part of Bhutan. Then came the clash in the Galwan Valley, during a June night in 2020. “One of the men came back and told the others in his unit,” Ranade said, “and they went over and there was a showdown.” The Chinese, according to the intelligence official, “don’t want India to be the long arm of the United States in this region or to be an active part of things like the Quad, which brings you back to the border issue.
    By the time the P.L.A. 4, 2022, the Chinese have built a new P.L.A. The result, the former diplomat told me, is that the Indian government then can’t send that delegate because doing so would register approval of China’s position. India has its own anxieties about Indians living near the border: It worries that they might shift their allegiance to China. India’s government has its own anxieties about Indians living close to the border: It worries that they might shift their allegiance to China. He then realized the phone was showing Chinese time, which is two hours and 30 minutes ahead of Indian time there. Belatedly, the Indian government has responded to China’s xiaokang villages with a “vibrant villages” program, announced in April last year.
  • Summary 2
    The mountains are part of the Kailash Range, a chain of rugged peaks, the tallest of which reach 22,000 feet, beginning near Pangong’s southern bank and extending southeast for some 500 miles. At the Tibetan border, this approach has taken a more physical form, as China has built more than 620 new “xiaokang” — or “moderately well-off” — villages all along the Tibetan border. “They have some fears that because India continues to give safe refuge to the Dalai Lama, at some point the Dalai Lama could return to Tawang and use it as leverage to galvanize the Tibetan people to try and declare independence from China,” Grossman says. “I’ve been to those places,” a former Indian diplomat who now works for an international Buddhist organization in Delhi told me.
    One night in early December 2022, for instance, hundreds of Chinese troops attempted to breach, in four spots, a stone wall along a border ridgeline in the Yangtze plateau, located on an eastern stretch of the border in India’s easternmost state, Arunachal Pradesh, which China claims is a part of Tibet. Why Do India and China Keep Fighting Over This Desolate Terrain?
    One such example is the Quadrilateral Security Dialogue, or Quad, a group made up of India, the United States, Australia and Japan, which share the goal of preventing China from dominating the Indo-Pacific. I can’t tell you what it is, but it doesn’t look good to me.’” Instead, embassy officials permit entry by stapling an unstamped piece of paper to the passport.
    A senior Indian intelligence official I met with in Delhi last year explained that China’s hostility along the Line of Actual Control had two strategic objectives: diminishing India’s impact in its own backyard and tying down India’s military in order to weaken India’s broader geopolitical influence. Pankaj Saran, who served as India’s deputy national-security adviser from 2018 to 2021 and now runs NatStrat, which researches security issues, contrasts India’s self-assuredness on the international stage today with its diffident foreign policy of the 1980s. Because of the difficult terrain, the heights along the range were left unoccupied by both India and China after the 1962 war.
    According to Chinese state media, the two Tibetan women and their father had been the sole inhabitants of their town Yumai for a period of years until the mid-1990s; its population has since risen to more than 200. Arunachal Pradesh, like much of India’s northeast, is less developed than other parts of the country; many rural communities in remote areas live in relative isolation. “When there’s a delegation of Indian bureaucrats traveling to China, the Chinese embassy here in Delhi will not issue a stamped visa to the delegate who belongs to Arunachal Pradesh,” says the former diplomat now at the nonprofit Buddhist organization.
    Soldiers from each side routinely leave empty cigarette packets and beer cans behind as marks of territorial claim. “Today,” he says, “we have a government that believes we’ve been taking the hit for far too long.” Xi has also emphasized China’s commitment to realizing its long-held dream of “reunification” with Taiwan, which split from mainland China in 1949. China has published such maps before. Episodes like those in Galwan and Yangtze reflect an era of increased hostility between the two countries, which have generally maintained a peaceable, if strained, relationship in the decades since they fought a war in 1962.
  • Summary 3
    One night in early December 2022, for instance, hundreds of Chinese troops attempted to breach, in four spots, a stone wall along a border ridgeline in the Yangtze plateau, located on an eastern stretch of the border in India’s easternmost state, Arunachal Pradesh, which China claims is a part of Tibet. Bhutan has historically relied on India’s help to defend its borders, so when China started to build a road into Doklam in June 2017, Indian troops entered the area to stop that construction, and the two sides formed human walls that faced off against each other. Srikant Kondapalli, a professor of China studies at Jawaharlal Nehru University, recounted to me what he had learned from an Indian brigadier about how Indian forces attempted to gain a psychological advantage in the conflict.
    China’s aggressive stance along the border with India, Kondapalli told me, is being driven by the same overarching goal of asserting sovereignty over disputed areas. A senior Indian intelligence official I met with in Delhi last year explained that China’s hostility along the Line of Actual Control had two strategic objectives: diminishing India’s impact in its own backyard and tying down India’s military in order to weaken India’s broader geopolitical influence.
    That gives the settlers in these villages an opening to encroach upon land that belongs to India, Kondapalli says. Indian authorities see the establishment of these border villages as buttressing a strategy of gradual encroachment — or “salami slicing,” as it’s known among security strategists — that China has practiced over the years in the South China Sea and is now attempting to replicate along the Line of Actual Control. A government spokesman said such actions by China served only to “complicate the resolution of the boundary question.”
    India sees China employing similar tactics to try to pressure India on Arunachal Pradesh, which the Chinese government calls Zangnan. “They say the individual is welcome because his land is a part of China.” The result, the former diplomat told me, is that the Indian government then can’t send that delegate because doing so would register approval of China’s position. India has its own anxieties about Indians living near the border: It worries that they might shift their allegiance to China. One reason for China’s interest in Arunachal Pradesh, especially its district of Tawang, is the existence of Buddhist holy sites in the state, including the Tawang monastery.
    Over the years, outreach efforts by Indian authorities have helped change that, he says: “Now, they understand that India is their country.” This fledgling, still-forming sense of Indian identity in parts of the border population is another reason the xiaokang villages built by China are a source of concern for the Indian government. He then realized the phone was showing Chinese time, which is two hours and 30 minutes ahead of Indian time there. Ngawang Tashi, a Buddhist monk from Arunachal Pradesh, told me that China’s attempts to woo India’s predominantly Buddhist border population is part of a larger effort to “sinify” Buddhism — that is, dilute its Tibetan identity and make it more Chinese.
  • Summary 4
    The earth is sandy, strewn with rocks and pebbles; not a blade of grass grows anywhere; there are no visible signs of animal life. India had to accept that Aksai Chin, an area of 15,000 square miles that it claimed as its own, would remain under Chinese control. (The Indian Army has not officially released any information about the operation, but I got a summary description of it from Gokhale.) The next thing they’ll do is, ‘The trails are not good enough, let’s start making roads.’ “It’s very simple, but very clear,” he told me over breakfast on the patio of a Delhi hotel.
    The brutal fighting in Galwan didn’t strike Ranade as entirely unexpected. “Back then, we were literally riding the coattails of the Soviet Union,” he told me. The action was “well planned, well thought out and executed, achieving total surprise,” Joshi said in a video interview with Nitin Gokhale, a veteran Indian military journalist who runs a foreign-affairs website called Stratnews Global. Then, they start making trails for the herders. When I visited Delhi last fall, the mood in the capital over Canada’s allegations was one of defiance. “We have to have parliamentary approval, this approval, that approval.”
    The Tawang monastery was the Dalai Lama’s first refuge in India when he fled Tibet in 1959, crossing over into Arunachal Pradesh after an arduous trek through the mountains. The Indian government denied the charge and demanded to see evidence of that claim. Under Xi, the country has converted coral reefs and sand piles dredged up from the seabed in the South China Sea, which it maintains belongs almost entirely to China, into artificial islands that are now heavily militarized with missiles and air strips. According to Indian press reports — the Indian Army has not provided a public account — the Chinese troops were armed with nail-studded clubs, monkey fists (knotted-up portions of rope used as a weapon) and stun guns.
    The high-ranking Indian intelligence official I spoke with in Delhi explained to me how the Chinese military had been operating on the border. But as the world’s fifth-largest economy, India no longer has any reason to be timid. China claims Doklam as its territory, while India and Bhutan maintain that the area is a part of Bhutan. A similar priority is the consolidation of Chinese control over Tibet by squashing a decades-long Tibetan independence movement. The violence in mid-June began when Col. The official went on: “Once the troops start coming in for patrols, then they’ll pitch tents, saying, ‘Our troops need to rest.’ “I’ve been to those places,” a former Indian diplomat who now works for an international Buddhist organization in Delhi told me.
0 voters

Comprehension of Summaries

In Example 2, you read summaries of the article 'Pooping on the Moon is a Messy Business.' Which option best describes the article?

  • The article discusses the issue of human waste management in space, particularly the disposal of feces by Apollo astronauts on the moon. It explores the potential environmental impacts, the technological challenges, and the cultural significance of managing waste in future lunar missions. It also touches on the broader implications for waste management on Earth.
  • The article focuses on the dietary habits of astronauts and the nutritional challenges they face during long-term space missions. It emphasizes the need for balanced meals and the role of different food types in maintaining astronaut health.
  • The article examines the psychological effects of isolation on astronauts during space missions. It highlights various coping mechanisms and the importance of mental health support for astronauts in prolonged space travel.
  • The article details the scientific experiments conducted on the moon by Apollo astronauts, focusing on geological surveys and the collection of lunar rock samples. It discusses the discoveries made about the moon's composition and surface features.
0 voters

Summary quality

Which summary has the best quality?"

  • Tiger
  • Pooping on the Moon is a messy business
  • Why Do India and China Keep Fighting Over This Desolate Terrain?
  • Apple
0 voters

Which summary has the worst quality?"

  • Tiger
  • Pooping on the Moon is a messy business
  • Why Do India and China Keep Fighting Over This Desolate Terrain?
  • Apple
0 voters

Of the all summaries in each articles you selected as best, how do you rate their quality?"

  • Yes, they are clear and informative.
  • Yes, but they could be improved.
  • No, they are unclear.
  • No, they are missing important information.
0 voters
3 Likes

Thank you for putting this survey together. Here's some initial feedback:

When article or unrelated link titles are included in these summaries, they appear out of place.

See the unrelated link title in most of the moon article summaries. That text was not found on the page when I read the article.

Also, see the article title embedded in the second India/China article summary.

I consider ideal summary length dependent on article size and user wants. Both can be unique for multiple or all summaries. If the question is to decide on a default length, I say pick the most performant option.

2 Likes

Hi, thank you for your useful feedbacks!!

I have just made some changes so it is now more readable, less confusing and suffocating.

1 Like

Thanks for creating this survey Hahabill. I feel it's a bit hard to participate though because one would have to read the Wikipedia article then compare the four options, which are lengthy and all a bit similar.

I don't have much suggestions for improvements though - maybe use shorter examples? But I don't know if that would help in your research and even then it would still be difficult to compare similar summaries.

Ironically I feel that comparing these samples and finding the best one would be something an IA would be a better at : )

(Also why is one of the options "Apple"? Is that to disqualify the users who didn't read the summaries?)

1 Like

There was a summary on an article about Apples like the others but it seems to have been deleted in an edit... @HahaBill are you aware that it seems to have gone AWOL?

1 Like

Hahah right, I could have just ask multiple LLMs for selecting the best option.

I picked mainly long complex articles to challenge the AI algorithms. However, I could make shorter summaries though.

To clarify why the "Apple" option is in one of the questions is because I had a section called "Summary Length" where I showed 4 summaries for "Apple" wikipedia article. Eventually, I have decided to remove the section since users can control the length of summaries outputted by the AI algorithms.
I did not want to remove the option since two people already voted so I just left with there. As you said, we can use that to disqualify the users who didn't read the summaries (I hope they will read the post after completing the survey hahaha).

1 Like

Yeah, I was aware. I explained that also in the reply to Laurent. I just decided to delete it since plugin provide users an option to choose the length of summaries.

With 3 participants, I already see interesting results.

1 Like

Unfortunately, I cannot change anything in the poll after 5 minutes since its initialisation :frowning:

To preface, I would like to hear thoughts on the below first, before anyone takes action on it. I'm aware that there is a time and effort crunch with GSoC involved.

I think the overall issue is using this Discourse feature over a shareable offsite survey platform, like in BibTeX Plugin: Survey.

Joplin Discord, subreddit, and other social media users should be allowed and encouraged to take the survey and provide their feedback.

Even then, a survey like this can be completed in good faith by those unaware of Joplin. Partly to Laurent's point, those could include IA, AI, and notetaking people. They may understand what makes for the best summaries better than we do.

I think using a survey platform as above, combined with shorter articles and summaries, would yield many more participants and thus better results.

1 Like

I see. I can post this survey in the GSoC Discord channel and see if the engagement increases. But it is a good idea to make the survey available for non-Discourse Joplin's users too!

Based on my observations, the problem with short articles is that the algorithms perform well on them. However, I can see more apparent differences between summaries from more prolonged and more complex articles or Wikipedia pages with multiple topics and parts. I don't know if there's added value to surveying much shorter articles. But I might be wrong.

But I agree that Example 3 - Why Do India and China Keep Fighting Over This Desolate Terrain? is difficult article to read and there are a lot of layers of the reason why they are fighting over the land/terrain.

I'll give this survey some more time and then I'll see.