616 lines
31 KiB
HTML
616 lines
31 KiB
HTML
<!doctype html>
|
|
<html>
|
|
|
|
<head>
|
|
<meta charset="utf-8">
|
|
|
|
<title>Xous: Rust Semantics in your OS</title>
|
|
|
|
<meta name="description" content="Overview of Xous, a microkernel with Rust semantics">
|
|
<meta name="author" content="Sean "xobs" Cross">
|
|
|
|
<meta name="apple-mobile-web-app-capable" content="yes">
|
|
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
|
|
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
|
|
|
|
<link rel="stylesheet" href="css/reveal.css">
|
|
<link rel="stylesheet" href="css/theme/fossasia2020.css" id="theme">
|
|
|
|
<!-- Theme used for syntax highlighting of code -->
|
|
<link rel="stylesheet" href="lib/css/zenburn.css">
|
|
|
|
<!-- Printing and PDF exports -->
|
|
<script>
|
|
var link = document.createElement('link');
|
|
link.rel = 'stylesheet';
|
|
link.type = 'text/css';
|
|
link.href = window.location.search.match(/print-pdf/gi) ? 'css/print/pdf.css' : 'css/print/paper.css';
|
|
document.getElementsByTagName('head')[0].appendChild(link);
|
|
</script>
|
|
|
|
<!--[if lt IE 9]>
|
|
<script src="lib/js/html5shiv.js"></script>
|
|
<![endif]-->
|
|
</head>
|
|
|
|
<body>
|
|
|
|
<!-- Start of main presentation -->
|
|
<div class="reveal">
|
|
<div class="footer">
|
|
<span class="theme">FOSSASIA Summit 2020</span>
|
|
<a class="url" href="https://p.xobs.io/fa20-bt/">p.xobs.io/fa20-bt</a>
|
|
<span class="hashtag">#fossasia</span>
|
|
<!-- <span class="twitter">@fossasia</span> -->
|
|
</div>
|
|
<div class="commentary"></div>
|
|
<div class="slides">
|
|
<section data-background-image="css/theme/lca2019-title-bg-transparent.svg">
|
|
<h2 style="background-color: transparent;">Betrusted: Being Secure</h2>
|
|
<h5 style="background-color: transparent;">That's no Blackberry, it's a chat client!</h5>
|
|
<p align="right" style="margin-bottom: 0px; margin-top: 0px; line-height: 1;">
|
|
<small>Sean Cross - <a href="https://xobs.io/">https://xobs.io/</a> - @xobs</small>
|
|
</p>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Betrusted: A Security Chip with I/O</h3>
|
|
<p>
|
|
<img data-src="img/bt-quarter2-shrunk.jpg" width="90%">
|
|
</p>
|
|
</section>
|
|
|
|
<section>
|
|
<section>
|
|
<h2>Xous: A Betrusted OS</h2>
|
|
<img data-src="img/tire_photo.jpg" class="fragment">
|
|
<aside class="notes">
|
|
A big question that gets asked is -- why another kernel? Why don't we just
|
|
put Linux on it and be done with it? Or something else like Minix, TockOS,
|
|
or FreeRTOS?
|
|
|
|
Much like this tire here, which is innovative in that it is airless, by
|
|
developing our own operating system we can achieve these goals, in addition
|
|
to creating some interesting new technology.
|
|
</aside>
|
|
</section>
|
|
|
|
<section>
|
|
<h2>Betrusted Goals</h2>
|
|
<ol style="width: 100%;">
|
|
<li><=4 MiB RAM</li>
|
|
<li>Safe language</li>
|
|
<li>Process Isolation</li>
|
|
</ol>
|
|
<ul style="text-align: left; width: 100%;" class="os-good">
|
|
<li class="fragment">Microkernel</li>
|
|
<li class="fragment">Auditable by one person</li>
|
|
</ul>
|
|
<aside class="notes">
|
|
With Betrusted, we wanted to reduce the code footprint. This will allow
|
|
us to run with less RAM -- ideally 4 MiB or less. We also wanted to have
|
|
a full MMU, which is somewhat unusual in the embedded microcontroller
|
|
space, where a more limited Memory Protection Unit is preferred.
|
|
|
|
Whereas the Linux kernel is huge, not to mention all of the support libraries
|
|
required to run a system, we would like the Betrusted system to be auditable
|
|
by one person. Lowering the memory footprint helps in this regard, as the
|
|
less RAM we have, the less code we must have.
|
|
|
|
Additionally, we would like to have the operating system written in a safe
|
|
systems language to protect us from common programming errors involving
|
|
memory and concurrency.
|
|
|
|
Even so, we would like to have full process isolation, so even if one process
|
|
is compromised, attackers will have a harder time boring through the system
|
|
to gain a more complete takeover. This allows us to have legacy software
|
|
written in non-safe languages, in case we need to take third-party code such
|
|
as font renderers from legacy systems.
|
|
|
|
As a result, we would like Betrusted to run a Microkernel-style operating
|
|
system, with "servers" that provide features such as the display,
|
|
keyboard, and even basic task switching. These should all run in
|
|
userspace with the bare minimum permissions required to get the job done.
|
|
</aside>
|
|
</section>
|
|
|
|
<section>
|
|
<h2>Microkernels</h2>
|
|
<img data-src="img/Annotation 2020-01-16 111040.png">
|
|
<div style="font-size: 12pt;">FlexSC: Flexible System Call Scheduling with Exception-Less System
|
|
Calls</div>
|
|
<aside class="notes">
|
|
Microkernels minimize the amount of code in each section. Everything from
|
|
the user-facing software to drivers run in userspace, with only memory
|
|
management and top-level exception dispatch taking place within the kernel.
|
|
|
|
By having a microkernel, individuals can contribute to servers, and be
|
|
responsible for their own section. This is not just one person working
|
|
on it, it enables lots of people to work on it together.
|
|
</aside>
|
|
</section>
|
|
|
|
<section>
|
|
<h2>Too Many Cooks</h2>
|
|
<table>
|
|
<tr>
|
|
<td style="width: 500px;">
|
|
<img data-src="img/code_ownership_effects_bugs.svg">
|
|
</td>
|
|
<td>
|
|
<blockquote class="os-quote" style="font-size: 24pt;">if there is one primary
|
|
contributor, <strong>the chances for a file to be buggy decreases
|
|
significantly</strong></blockquote>
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Source: <a
|
|
href="https://docs.microsoft.com/en-us/azure/devops/learn/devops-at-microsoft/code-ownership-software-quality">Microsoft
|
|
Research</a></td>
|
|
</tr>
|
|
</table>
|
|
<aside class="notes">
|
|
According to a Microsoft research paper analyzing the failures of Windows Vista,
|
|
the number one predictor for code quality is the number of people who work on
|
|
a module. If one person works on the project, then the number of bugs goes down.
|
|
</aside>
|
|
</section>
|
|
<section>
|
|
<h2>Felix' Rule of Thumb</h2>
|
|
<table>
|
|
<tr>
|
|
<td style="width: 380px;">
|
|
<img data-src="img/Cthulhu_sketch_by_Lovecraft.jpg">
|
|
</td>
|
|
<td>
|
|
<blockquote class="os-quote" style="width: 100%; margin-left: 0; margin-right: 0;">
|
|
The largest amount of security-related code that one person can reasonably audit is
|
|
about 64 KiB of binary data
|
|
</blockquote>
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
<aside class="notes">
|
|
A friend of ours named Felix has a rule of thumb: The barrier at which a codebase becomes
|
|
too much to reason about for one individual is about 64 KiB. Anything more than this and
|
|
it becomes an eldrich horror that morphs and changes when you're not looking at it. As
|
|
a result, we would like to keep the core of the system small, so that we can keep it in
|
|
our heads as we reason about the system.
|
|
</aside>
|
|
</section>
|
|
|
|
<section>
|
|
<h2>Principles of Software</h2>
|
|
<table width="100%">
|
|
<tr>
|
|
<td style="text-align: right">
|
|
<img width="75%" class="fragment"
|
|
data-src="img/Rust_programming_language_black_logo.svg">
|
|
</td>
|
|
<td valign="top" width="50%">
|
|
<ul style="margin-left: 0px;">
|
|
<li>Safety</li>
|
|
<li>Concurrency</li>
|
|
<li class="fragment fade-semi-out">Speed</li>
|
|
<li class="fragment">Size</li>
|
|
</ul>
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
<aside class="notes">
|
|
The Rust programming language promises the holy trifecta: Safety, Speed,
|
|
Concurrency. Pick any three. If you're going to start over on a systems-
|
|
level project, choose Rust. There will be a lot of wailing and gnashing
|
|
of teeth to begin with, but the end result will be better.
|
|
|
|
When we started Betrusted, we decided that it should primarily use Rust
|
|
as the systems language. That way we can be sure that our code is sound.
|
|
Additionally, Rust has the ability to produce efficient binaries, and the
|
|
efficiency is only going to get better as time progresses.
|
|
|
|
This eliminates non-Rust choices such as Linux or Minix.
|
|
</aside>
|
|
</section>
|
|
|
|
<section>
|
|
<h2>Rust OS Landscape</h2>
|
|
<img class="fragment" data-src="img/os-survey-stale.png">
|
|
<aside class="notes">
|
|
Having decided to use Rust, we did a survey of Rust-based operating
|
|
systems. There are several available, in various states of completeness.
|
|
Many projects have long since been abandoned, which is fine because up
|
|
until recently you needed to work on Rust nightly to build an OS. The
|
|
language underpinnings of these projects has shifted, and so many of them
|
|
have been abandoned. A few are still going, and
|
|
the two biggest candidates are Redox and Tock.
|
|
</aside>
|
|
</section>
|
|
|
|
<section>
|
|
<h2>Rust-based OS: Tock</h2>
|
|
<table>
|
|
<tr>
|
|
<td colspan="2" style="text-align: center;">
|
|
<img height="250px" data-src="img/tockos.svg">
|
|
</td>
|
|
</tr>
|
|
<tr style="font-size: 24pt">
|
|
<td>
|
|
<ul class="os-good fragment">
|
|
<li>Active Project</li>
|
|
<li>RISC-V Port</li>
|
|
<li>C and Rust Libs</li>
|
|
</ul>
|
|
</td>
|
|
<td>
|
|
<ul class="os-bad fragment">
|
|
<li>No MMU Support</li>
|
|
<li>No runtime spawn()</li>
|
|
<li>Limited messaging</li>
|
|
</ul>
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
<aside class="notes">
|
|
Tock is the most obvious choice, since it already has a RISC-V port
|
|
and is supported by a well-documented ABI. Tock supports multiple
|
|
tasks written in either Rust or C, which is a very nice feature.
|
|
|
|
However, Tock does not support an MMU. It would be possible to adapt
|
|
the MPU interface to work with an MMU, but a lot of design work has
|
|
gone into Tock to make it work well with only the standard MPU that
|
|
is present on most ARM chips. Using Tock would be asking it to do
|
|
something that it's not designed for. Instead, it's better to pick
|
|
the right tool for the job.
|
|
|
|
Additionally, the Tock message passing infrastructure assumes only
|
|
one server per process, which can limit flexibility.
|
|
</aside>
|
|
</section>
|
|
|
|
<section>
|
|
<h2>Rust-based OS: Redox</h2>
|
|
<table>
|
|
<tr>
|
|
<td colspan="2" style="text-align: center;">
|
|
<img height="250px" data-src="img/Redox_logo_2015.svg">
|
|
</td>
|
|
</tr>
|
|
<tr style="font-size: 24pt">
|
|
<td>
|
|
<ul class="os-good fragment">
|
|
<li>Active Project</li>
|
|
<li>Full Rust stdlib</li>
|
|
<li>Full Userspace</li>
|
|
</ul>
|
|
</td>
|
|
<td>
|
|
<ul class="os-bad fragment">
|
|
<li>x86_64 only</li>
|
|
<li>Unix-like</li>
|
|
<li>Desktop-focused</li>
|
|
</ul>
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
<aside class="notes">
|
|
Redox is the other large Rust-based operating system, and is the most
|
|
compelling. Redox is currently limited to x86-64 (with a plan to
|
|
port it to AArch64), so we would need to port it ourselves to RISC-V.
|
|
|
|
The biggest problem with Redox is its size: It's a full Desktop
|
|
operating system, and supporting it on Betrusted would require us to
|
|
spend a lot of time cutting it down to just the bare microkernel,
|
|
at which point we can start to recreate everything ourselves. The
|
|
kernel itself is such a small part of Betrusted. Besides, we would
|
|
like the freedom to experiment, to randomize the syscall numbers and
|
|
have keepout areas of the screen and add IMEs to the input, which
|
|
would quickly introduce incompatibility with Redox.
|
|
|
|
Finally, we would like to be able to use stable Rust for our applications,
|
|
which we can almost do with Xous. We're just waiting for either the
|
|
"alloc_error_handler" attribute to be stabilized (issue 66740), or
|
|
defaulting handle_alloc_error to panic (issue 66741), which would give
|
|
us everything we'd need to use alloc on stable Rust.
|
|
|
|
In short, Tock is too small, and Redox is too big.
|
|
</aside>
|
|
</section>
|
|
|
|
<section>
|
|
<h2>Rust-based OS: Tifflin</h2>
|
|
<table>
|
|
<tr>
|
|
<td colspan="2" style="text-align: center;"><span style="font-size: 168pt">?</span></td>
|
|
</tr>
|
|
<tr style="font-size: 24pt">
|
|
<td>
|
|
<ul class="os-good fragment">
|
|
<li>Active Project</li>
|
|
<li>Rust stdlib</li>
|
|
<li>Full Userspace</li>
|
|
</ul>
|
|
</td>
|
|
<td>
|
|
<ul class="os-bad fragment">
|
|
<li>nightly only</li>
|
|
<li>Mainly x86_64</li>
|
|
<li>???</li>
|
|
</ul>
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
<aside class="notes">
|
|
Tifflin is a kernel that's been around for a while, but I've
|
|
only just learned about. It has a lot of promise, and seems
|
|
to be an interesting desktop operating system. I must admit
|
|
I don't know much about it, because it's not well-publicised.
|
|
However, it does have a rust stdlib, the design of which we
|
|
may borrow for Xous. One downside to it is that it requires
|
|
the nightly compiler, whereas we want to focus on stable for Xous.
|
|
</aside>
|
|
</section>
|
|
|
|
<section>
|
|
<h2>Other Alternatives and Inspirations</h2>
|
|
<ul>
|
|
<li><strong>ChibiOS</strong> - Embedded RTOS</li>
|
|
<li><strong>HelenOS</strong> - Everything is a message</li>
|
|
<li><strong>Solaris</strong> - Doors</li>
|
|
<li><strong>QNX</strong> - Traditional Microkernel</li>
|
|
</ul>
|
|
<div>Microkernels isolate and make IPC cheap</div>
|
|
<aside class="notes">
|
|
There are many alternative operating systems. We can draw inspiration
|
|
from them, even if we don't use them directly.
|
|
|
|
For example, both the QNX microkernel and Solaris Doors implementation
|
|
allow for one process to pass a message to another, which then inherits
|
|
its remaining quantum and runlevel. This prevents priority inversions
|
|
and makes syscalls relatively cheap.
|
|
</aside>
|
|
</section>
|
|
</section>
|
|
|
|
<section>
|
|
<section>
|
|
<h2 style="margin-top: 25%;">Xous: System Design</h2>
|
|
<aside class="notes">
|
|
Xous is, currently, very much under development. However, there has
|
|
been a lot of planning.
|
|
</aside>
|
|
</section>
|
|
|
|
<section>
|
|
<h2>Xous: Memory Model</h2>
|
|
<table>
|
|
<tr>
|
|
<td>
|
|
<img data-src="img/Library_card.jpg">
|
|
</td>
|
|
<td height="100%">
|
|
<ul class="os">
|
|
<li>Rust Borrow Checker</li>
|
|
<li class="fragment">Message passing</li>
|
|
<li class="fragment" style="font-weight: 500">Inter-process borrowing</li>
|
|
<li class="fragment">Borrow types:
|
|
<ul>
|
|
<li class="fragment">Mutable ^ Immutable</li>
|
|
<li class="fragment">No Access | Read Only</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
<a style="font-size: 12pt;" href="https://www.flickr.com/people/9337414@N05">Image CC-BY Tammy</a>
|
|
<aside class="notes">
|
|
Xous will base its memory model on the Rust borrow checker. That is,
|
|
shared memory will be used for IPC. If one process wishes to get a
|
|
response from another, it can pass pages via a mutable borrow. If
|
|
a process wishes to share pages across multiple process, then only an
|
|
immutable borrow may be made, and the sharing process cannot access
|
|
pages until all processes have returned the memory.
|
|
|
|
A process can move memory into another, which for example is how
|
|
process spawning works. In such a case, memory is no longer available
|
|
in the sending process.
|
|
</aside>
|
|
</section>
|
|
|
|
<section>
|
|
<h2>Xous: Memory Model</h2>
|
|
<ol>
|
|
<li class="fragment">Mutable Borrow
|
|
<ul>
|
|
<li>draw()</li>
|
|
</ul>
|
|
</li>
|
|
<li class="fragment">Immutable Borrow
|
|
<ul>
|
|
<li>Mapping font database</li>
|
|
</ul>
|
|
</li>
|
|
<li class="fragment">Move
|
|
<ul>
|
|
<li>Encrypting a string</li>
|
|
</ul>
|
|
</li>
|
|
</ol>
|
|
</section>
|
|
|
|
<section>
|
|
<h2>Xous: Interrupts</h2>
|
|
<pre><code style="font-size: 23pt; line-height: 1.2em;" class="rust">fn setup_int2() -> xous::Result {
|
|
let gpio = xous::syscall::map_physical(
|
|
Some(0xe0000000), None, 4096)?;
|
|
|
|
xous::syscall::claim_interrupt(2, |_int_num, gpio| {
|
|
unsafe {
|
|
let val = gpio.read_volatile();
|
|
gpio.write_volatile(val + 1);
|
|
};
|
|
}, gpio)
|
|
}</code></pre>
|
|
<div class="fragment">All in userspace</div>
|
|
<aside class="notes">
|
|
Interrupts block the whole system, and follow similar behavior to
|
|
memory. That is, each interrupt can only be assigned to a single
|
|
handler. This is an example of a server claiming interrupt 2,
|
|
and this function will be called to handle that interrupt. It will
|
|
be called in Supervisor mode with the process space of this server.
|
|
During the interrupt handler, interrupts are disabled, and after it
|
|
returns they will be re-enabled. You can't make any syscalls in
|
|
this mode that don't have an "_i" suffix. This will form the
|
|
basis of drivers running in userspace using safe code.
|
|
</aside>
|
|
</section>
|
|
|
|
<section>
|
|
<h2>Xous: Missing Features</h2>
|
|
<table style="width: 100%">
|
|
<tr>
|
|
<td style="width: 300px">
|
|
<img data-src="img/missing-image.svg">
|
|
</td>
|
|
<td>
|
|
<ul class="os-bad">
|
|
<li class="fragment">fork()</li>
|
|
<li class="fragment">Filesystem</li>
|
|
<li class="fragment">Scheduler</li>
|
|
<li class="fragment">Threads</li>
|
|
<li class="fragment">Locking primitives</li>
|
|
<li class="fragment">Shared libraries</li>
|
|
</ul>
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
<aside class="notes">
|
|
Xous has no fork(). Instead it will have spawn().
|
|
|
|
Xous has no scheduler. The scheduler will be implemented as a userspace
|
|
program, which will request the Timer interrupt and call a kernel function
|
|
to preempt the current process.
|
|
|
|
It has no threads. However, there is enough information passed from the
|
|
kernel to enable the userspace scheduler to implement threads.
|
|
|
|
Similarly, it has no kernel-level locking primitives. Because memory can't
|
|
be shared between processes, there is no need for inter-process locking.
|
|
Within a process, threads are available with LLVM intrinsics such as cmpxchg.
|
|
|
|
Shared libraries aren't available at the start, but may come later via
|
|
shared immutable borrows from the linker server.
|
|
</aside>
|
|
</section>
|
|
|
|
<section>
|
|
<h2>Xous: Everything in Userspace</h2>
|
|
<ul style="width: 100%;">
|
|
<li>Small Kernel</li>
|
|
<li>Message Passing</li>
|
|
<li>Protected Memory</li>
|
|
</ul>
|
|
<br />
|
|
<br />
|
|
<div><strong class="fragment" style="font-size: 40pt;">Understandable by one human</strong></div>
|
|
<div><strong class="fragment" style="font-size: 40pt;">Made by many</strong></div>
|
|
</section>
|
|
</section>
|
|
|
|
<section>
|
|
<section>
|
|
<h2>Developing Xous</h2>
|
|
<aside class="notes">
|
|
Xous is developed using Renode, which is an amazing
|
|
emulation platform. It's cross-platform, and is very
|
|
easy to get started with.
|
|
</aside>
|
|
|
|
</section>
|
|
</section>
|
|
</div>
|
|
</div> <!-- class="reveal" -->
|
|
<!-- End of main presentation -->
|
|
|
|
<!-- Start of configuration section -->
|
|
<script src="lib/js/head.min.js"></script>
|
|
<script src="js/reveal.js"></script>
|
|
|
|
<script>
|
|
var presenter = !!Reveal.getQueryHash().s;
|
|
|
|
// More info https://github.com/hakimel/reveal.js#configuration
|
|
Reveal.initialize({
|
|
controls: presenter ? false : true,
|
|
progress: true,
|
|
history: true,
|
|
center: false,
|
|
controlsTutorial: presenter ? false : true,
|
|
|
|
slideNumber: presenter ? null : 'c/t',
|
|
|
|
// The "normal" size of the presentation, aspect ratio will be preserved
|
|
// when the presentation is scaled to fit different resolutions. 16:9 is common.
|
|
width: 1280,
|
|
height: 720,
|
|
|
|
// Factor of the display size that should remain empty around the content
|
|
margin: 0.1,
|
|
|
|
multiplex: {
|
|
url: 'https://p.xobs.io/',
|
|
id: 'd03979a76e514b4c',
|
|
secret: Reveal.getQueryHash().s || null
|
|
},
|
|
|
|
// Bounds for smallest/largest possible scale to apply to content
|
|
minScale: 0.02,
|
|
maxScale: 5.5,
|
|
|
|
transition: 'slide', // none/fade/slide/convex/concave/zoom
|
|
|
|
// More info https://github.com/hakimel/reveal.js#dependencies
|
|
dependencies: [
|
|
{ src: 'lib/js/classList.js', condition: function () { return !document.body.classList; } },
|
|
{ src: 'plugin/markdown/marked.js', condition: function () { return !!document.querySelector('[data-markdown]'); } },
|
|
{ src: 'plugin/markdown/markdown.js', condition: function () { return !!document.querySelector('[data-markdown]'); } },
|
|
{ src: 'plugin/highlight/highlight.js', async: true, callback: function () { hljs.initHighlightingOnLoad(); } },
|
|
{ src: 'plugin/search/search.js', async: true },
|
|
{ src: 'plugin/zoom-js/zoom.js', async: true },
|
|
{ src: 'plugin/notes/notes.js', async: true },
|
|
|
|
{ src: 'lib/js/socket.io.js', async: true },
|
|
{
|
|
src: presenter ?
|
|
'plugin/multiplex/master.js' :
|
|
'plugin/multiplex/client.js', async: true
|
|
},
|
|
]
|
|
});
|
|
|
|
// After the talk, show speaker notes on the slide.
|
|
if (new Date() > new Date(1584770026751)) {
|
|
Reveal.addEventListener('slidechanged', function (event) {
|
|
// event.previousSlide, event.currentSlide, event.indexh, event.indexv
|
|
var commentaries = document.getElementsByClassName("commentary");
|
|
Array.prototype.forEach.call(commentaries, cmt => {
|
|
cmt.innerHTML = "";
|
|
cmd.style.display = "none";
|
|
});
|
|
event.currentSlide.childNodes.forEach(element => {
|
|
if ((element.nodeName == "ASIDE") && (element.className == "notes")) {
|
|
Array.prototype.forEach.call(commentaries, cmt => {
|
|
cmd.style.display = "";
|
|
cmt.innerHTML = "<p>" + element.innerHTML.replace("\n\n", "</p><p>") + "</p>";
|
|
});
|
|
}
|
|
});
|
|
});
|
|
}
|
|
</script>
|
|
</body>
|
|
|
|
</html>
|