Index
- Annotation
- Bubble
- Correction of science
- Critical citations
- Epistemic errors
- Hype
- Norms, normative
- Ontology
- Post-publication peer review
- Precision & accuracy
- Replicability/Reproducibility/Repeatability
- Representation
- Research integrity (+ relation to ethics)
- Scientific claim & counter-claim disagreement
Annotation (Federico Boem, Martin Lentschat)
Annotation for NLP
- build Gold Standards for training and evaluation
- based on a framework (e.g. entity tagging accordingly to a controlled vocabulary)
- framework is task dependant
Other methods use lexical resources (e.g. dictionary, thesaurus) to automate the annotation process, fully or partially.
Annotation for peer review
- Keeping track and share insight
Annotation refers to a critical reading of the content and structure of a text. The annotator then reports, classifying the types of comments according to different types of problems, any logical inconsistencies, incorrect or unclear use of language, methodological issues, conceptual errors (e.g. showing how the experimental techniques used do not adequately respond to the scientific hypothesis being investigated) or in the representation of the data.
________________________________________________________________________________
Bubble (Cyrus Mody, Max Roßmann)
- A (physical) phase-interface
- e.g., a soap bubble
- A bursting bubble of hope & investments
- Who and how is hot air blown into a bubble?
- Don’t overstretch the promises or the bubble may burst! (or develop in something else?)
- Perquisites for economic bubbles (Goldfarb & Kirsch 2019):
- Potential for promissory stories & narrative accelerators
- Uncertain success of technology functioning or its commercialization
- Tightly coupled and specialized “pure play” firms
- Free capital or inexperienced investors
- Examples: AM radio (1929), Personal aviation (1929), office software (1984), online shopping (1999)
- A filter bubble OR echo chamber (Nguyen, 2020)
- within the bubble you do not hear (epistemic bubble: no facts)
- OR not acknowledge other’s opinions (echo chamber: no trust).
Project Name “NanoBubbles“:
Paper: “Is Nano a Bubble?” (Möhwald & Weiss 2015; focus: a bursting bubble?) → Project: “NanoBubble” (Seed Meeting, 29/05/2019) –> NanoBubbleS
________________________________________________________________________________
Correction of Science (Federico Boem)
The sciences are a collective enterprise, organised according to certain methodological principles (which are, however, often discussed and renegotiated) as well as certain societal values (such as transparency, integrity, fairness, rigour, elegance etc.). The forces conditioning scientific progress can be both internal to disciplinary practices, such as methodological conventions, and external, such as economic and political interests.
Scientific knowledge is supposed to have a certain epistemic robustness, i.e. it possesses the ability to resist attempts to be refuted, which is considered superior to other types of knowledge.
The ability of science to self-correct is considered one of the hallmarks of scientific research, the process or manner by which sciences are corrected can be difficult to define. Correction, or sometimes more accurately the correcting of science, can be about changing the literature (e.g. redactions), challenging the values that underpin scientific research, or scientific methods, among other approaches.
The possibility of self-correction, however, reveals the revisable and updateable character of scientific knowledge itself. The boundaries of correction are therefore fuzzy, and some changes might not count as correction. For instance, when scientific knowledge is updated, or incrementally modified, or in cases where new knowledge supersedes old, this may be done without being considered ‘correction’.
Self-correction also shows how, with regard to the contents it produces, the sciences (in principle) recognise no authority other than science itself (represented by the community of researchers) as the final arbiter and judge of the cognitive value of these contents. But these communities can and do include more than just individual scientists, i.e. institutions, organisations, funders, industry, and lay people can also be considered stakeholders.
________________________________________________________________________________
Critical citations (Frédérique Bordignon)
A citation is the explicit reference to another scientific piece of work within the full-text of a scientific publication. With a critical citation, authors carry a negative connotation towards to the work they cite.
- To criticize: to point out a weakness or a fault in the cited work. Chiang (2005) introduced a constituent feature to reward phrases that match a syntactic tree but did not yield significant improvement.
- To question: to convey concerns, doubts, and uncertainty about the cited work. Note that this calls into question analyses of agreements based on a single coder’s tagging effort, e.g. Walker 1996.
- To compare: to express that one study is better than another, without necessarily including one’s own work (it can be two other publications). Our results appear to contradict those of [35].
Example: However, not all authors agree that treating this disease reduces concentrations in serum (|B24|) → non-critical!
________________________________________________________________________________
Epistemology is the study (or philosophy) of knowledge, which encompasses many theories about what knowledge is. For instance, knowledge isn’t just what is considered ‘true’ or ‘factual’ since there are also elements of belief in what people know, or think they know. This is why knowledge has sometimes been described as ‘justified, true belief’, though this is also contested. Whatever the boundaries we set, it’s clear that knowledge is not neutral. What we think of something as knowledge we need to consider that this will include a whole set of values, norms, and social expectations, among other things. There are many theoretical approaches to knowledge, including rationalism (knowledge through reason) and empiricism (knowledge through the senses). There are also contemporary challenges to some dominant approaches, e.g. standpoint theory, which points to subjectivity in knowledge. There are also plenty of critical approaches that highlight ambiguity, uncertainty, and irreducibility in knowledge building. Errors can be considered as mistakes, whether in perception, judgement, data, or reporting, as well as those that happen at more fundamental levels, e.g. in reasoning and argument (see also fallacies). As Russell (1912) notes, “a true belief is not knowledge when it is deduced from a false belief”. From all this we can deduce that an epistemic error is a failure in the content of what we know (e.g. from the method or data) and/or in the process of knowing something (e.g. from reasoning, argumentation, or perspective).
________________________________________________________________________________
Hype and overpromising (Stefan Gaillard, Max Roßmann)
Hype
Hyping means inappropriately exaggerating benefits (or worries –> alarmism). Exaggerations are inappropriate when they hinder communication aims, such as accuracy or predictive relevance, e.g., for the sake of enthusiasm or attention, or when they suggest harmful but unsupported inferences, .e.g., implicitly by omitting information. As value judgement, hype is relative to communication context, audience, and situation.
Hype is empirically studied a hyperbolic curve of attention, hope, and optimism over time (= a rise followed by a decline). Popular indicators are publication counts, stock market trends, optimistic over critical representations, hyping language, and other sentiment indications.
Hype is studied as explanandum of ignorance or violated communication norms and as explanans for malicious consequences, such as misled decisions, wrong medical treatments, wasted resources, ignorance of perusing other solutions, or a general loss of trust in science.
Overpromising
- Normative vs. epistemic dimension:
- Overpromising is different from dishonesty and fraud, yet still has a normative dimension.
- Rather, overpromising is an epistemic mistake.
- Logical vs. physical possibility:
- Logically impossible promises;
- Physically impossible promises;
- Improbable promises.
- Explicit vs. implicit claims about the future
________________________________________________________________________________
Norms, normative (YJ Erden, Stefan Gaillard)
A ‘norm’ (in philosophy) captures either a standard or convention that is expected or promoted. For example, norms can refer to social, ethical, legal or technical standards and conventions.
Norms can reflect values which may overlap or be in tension. For example, traditional scientific norms reflecting trust in science may be in tension with modern norms reflecting replicability & transparency.
‘Normative’ captures the establishment, promotion, defence, or regulation of certain norms. It prescribes how things ought to be, and can include rules to reify these, whether explicit or implicit. Normative behaviours or statements can be prescriptive and value-laden, for example, “the scientific method is valuable”.
The outcomes and impacts of transgressing norms (like rules), or challenging normative approaches (like cultural etiquette), can include punishment, exclusion, or censure, whether direct or indirect.
________________________________________________________________________________
Ontology (Federico Boem, Martin Lentschat)
Ontology (philosophy)
Ontology refers to that part of philosophy that investigates what there is (or what exists)
An ontological question can be specific:
- e.g. does God exist / does an electron exist, and would such existence be of the same kind as macroscopic objects in our everyday experience (like chairs)
- e.g. how is a hand ‘part of’ a body or water droplets ‘part of’ a river
Ontological questions can also be broader:
- e.g. general characteristics and relations of entities that exist, like the relation between the concept ‘river’ and the river Seine.
Ontology (knowledge engineering)
A “data framework” using concepts and relations to represent domain knowledge
- Nodes: individuals, classes, attributes.
- can include a vocabulary
- Edges: subsumption or semantic relations
Building an ontology:
- For what we are going to use the ontology?
- For what types of questions the information in the ontology should provide answers?
- Who will use and maintain the ontology?
Ontologies are used in semantic web, information retrieval, expert systems, AI…
________________________________________________________________________________
Post-publication peer review (Nathanne Rost)
- Critical analysis is the analysis of the substantion of the claims by the results and conclusions presented; evalutation of the described methodology (consistency, details provided) and in some cases, of the cited literature.
- Commenting leads to organising meetings among the annotators in order to discuss the points we consider important. Then, we collaboratively write the comment.
- By posting the comment, we share the outcomes of the analysis (e.g. by posting comments on PubPeer and writing letters to the editors).
- Most common flaws are:
- image duplication
- lack of information in the methods
- miscitation
- conclusions not supported by the results
- overpromises
You can read more about PPPR on our dedicated webpage.
________________________________________________________________________________
Precision & accuracy (YJ Erden, Martin Lentschat)
|
Elements (N) |
Retrieved |
Discarded |
|
Positive |
True Positive (TP) |
False Negative (FN) |
|
Negative |
False Positive (FP) |
True Negative (TN) |
- Precision:
- Proportion of valid elements among the retrieved
- TP / (TP + FP)
- Recall:
- Proportion of retrieved elements among the relevant ones
- TP / (TP + FN)
- Accuracy:
- Correct predictions among all elements
- (TP + TN) / N
“Accuracy is the closeness of a measurement to the true value of what is being measured. Precision is the closeness of repeated measurements to each other.” Bendick, R., & Borgmann, A. (2017). Explanation in philosophy and the limits of precision. AI & society, 32, 167-174.
“The distinction between the terms accuracy and precision – terms which are mistakenly used interchangeably – corresponds to this dichotomy between systematic and random errors. Accuracy refers to the closeness of the measurements to the ‘true’ value of the sought physical quantity, whereas precision indicates the closeness with which the measurements agree with one another independently of their relations to the ‘true’ value. Accuracy thus implies precision but the converse is not necessarily true.” Hon, G. (1989). Towards a typology of experimental errors: An epistemological view. Studies in History and Philosophy of Science Part A, 20(4), 469-504.
________________________________________________________________________________
Replicability/Reproducibility/Repeatability (Maha Said)
Replicability
The closeness of agreement between independent results obtained with the same method on same test material but under different conditions (different team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same or a different location) on multiple trials.
Reproducibility
The closeness of agreement between independent results obtained with the same method on identical test material but under different conditions (different operators, different apparatus, different measuring systems different laboratories and/or after different intervals of time).
Repeatability
The closeness of agreement between independent results obtained with the same method on identical test material, under the same conditions (same apparatus, same laboratory and after short intervals of time) multiple preparations of the same sample.
Definitions are adapted from IUPAC. Compendium of Chemical Terminology, 2nd ed.
________________________________________________________________________________
Object (-features) represent a target in a rule-governed practice.
- Metaphors & representations are not necessary reflexive & not similar to their target:
- A metaphor denotes something (only) in a particular way.
- Features of a representation (a map, image, text, or model) denote features of a target system in a rule-governed practice (a key to instruct imagination / suspend disbelief).
- Both affect and build on cognitive & cultural structures to be deciphered.
- Culture: We must fill gaps to understand representations (explicit / implicit f-truth)
- Power: Representations can be (in-)official / (un-)authorized within a community / culture.
- Objects (or a set of objects) can suit to represent for different purposes:
- A map suits to orient in a city or hunt a treasure; a graph or equation suits to design an experiment.
- A narrative suits to merge potential actions and events to a motivating research endeavour.
- Representations can (mis-)match beliefs, desires, or other representations:
- A narrative, artwork, or model matches our beliefs about reality only in some ways but mismatches in others.
→ context, power & purpose matter for (mis-)understanding metaphors & representations. We must reflect about:
- how literally we should understand, e.g., the “computation” metaphor to learn about gene regulation or human cognition.
- what image features help to generate attention or motivate collaborations but jeopardize certain learnings.
________________________________________________________________________________
Research integrity + relation to ethics (Federico Boem, YJ Erden)
Research integrity generally refers to a set of ethical principles and values, deontological duties and professional standards. These principles and standards underpin responsible and correct (e.g. acceptable or otherwise) conduct by those who undertake, fund, or evaluate scientific research. They apply to individual researchers as well as to institutions.
The application of principles, values and standards contribute to guarantees of research quality. They also contribute to preserving and enhancing trust and reliability. Integrity is typically used as a tool to protect the public image of science, which is essential to advancements and progress.
Integrity and ethics are related but one is not reducible to the other! It is sometimes easier to establish rules for integrity compared to agreeing ethical rules.
________________________________________________________________________________
Scientific claim & counter-claim disagreement (Willem Halffman, Raphaël Lévy)
- Analytically, scientific claims: statements about what is the case, i.e. deductive or inductive inferences.
- Explicit claims: articulated in language or graphs;
- Implicit claims: unstated, relied on in the argument
- e.g: on surface density of DNA: “In the case of spherical particles, smaller particles can support higher densities, substantially greater than values obtained on planar surfaces.” (Cutler ea 2012). The explicit claim of higher DNA density presupposes implicit claims such as: particles can have DNA ligands, density is measurable and comparable, DNA exists,…
-
- Virtually every sentence in an article is a statement about what is the case => every article contains about as many explicit claims as sentences.
- Scientific articles have main claims, the ‘point’ of an article, typically represented in the discussion/conclusion and/or abstract.
- These main points may only become clear or understandable when juxtaposed to other texts, e.g. other scientific articles.
- Explicitly: references or connections stated in the text
- Implicitly: relying on the reader’s knowledge
- However, in scientific writing/rhetoric, to call something a ‘claim’ is a subtle way toquestion a statement:
- ‘it is claimed that…’ suggests it is unproven or in question (i.e. open to doubt, needing evidence).
- “To test this claim, we prepared stable C166 cells that that have a deficient expression level of class B scavenger receptor type I, (…)”
(Choi, C H J ea, Mechanism for the endocytosis of SNA conjugates, PNAS, 2013)
- Calling statements in articles ‘claims’ can therefore be perceived as challenges to author’s findings.
- Counter-claims: statements about what is the case articulated in contrast with claims made by others: stating that something else is the case, or that the original claim is incorrect.
- “This set of results demonstrated that when considering entropic contributions, bcc is always favored over fcc, contrary to results regarding enthalpic contributions.” (Thaner ea, Entropy-driven Crystallization…, Nano Lett 2015)
- The reference to the original claim can be explicit or implicit, in which case it is left to the reader to understand or make the contrast.
- “This set of results demonstrated that when considering entropic contributions, bcc is always favored over fcc, contrary to results regarding enthalpic contributions.” (Thaner ea, Entropy-driven Crystallization…, Nano Lett 2015)
