Building a Webmention Receiver and Sending Webmentions
Tags: js protocols webmention post
Introduction
Today we’re going to build our own Webmention receiver which will allow us to receive webmentions. We’ll also go over how to send Webmentions so we can test out our own implementation. If you haven’t read my past post on Webmention endpoint discovery, I highly recommend you give it a read although it’s not necessary to understand this post.
So what exactly are webmentions? Webmention is a method to send comments, likes, or other web actions to other sites! It allows posters to retain ownership of what they wrote and host it on their own site, while at the same time allowing external websites to display that same data. This protocol is a standard which was developed by the IndieWeb community and now sponsored by the W3C. Essentially, a technical standard allows multiple different groups to create different implementations, while allowing them all to share a common base and to be interpolatable. The web is based off of standards which allow different browsers to display websites. You may notice that some websites may act a bit differently in different browsers. That’s because not every implementation implements the standard exactly the same, and sometimes there can be small deviations. We’re going to try to stick to the Webmention specification as closely as possible, but there are some areas where we could make some changes to be more in line with it. I’ll try my best to specify where that is.
Although as I go through this post I treat the specification as an absolute, it’s important to remember that webmentions are a living thing. Most of the basic “plumbing” of how a webmention is sent is relatively fixed. Yet, the ways webmentions are interfaced with, what exactly is sent and how it’s displayed, is changing and evolving. There are no limits with this technology so I encourage you to explore them.
Sending Webmentions
Before we can start building a Webmention receiver that can receive webmentions and process them, we first need a way to send webmentions. In this post, we’re going to work on sending Webmentions using JavaScript on Node.js. Below is our function that we can use to send webmentions:
const fetch = require('node-fetch');
var formurlencoded = require('form-urlencoded').default;
function sendWebMention(source, target, webmentionEndpoint, callback){
fetch(webmentionEndpoint, {
body: formurlencoded({source: source, target: target}),
headers: {
"Content-Type": "application/x-www-form-urlencoded"
},
method: "POST"
})
.then(res => res.text())
.then(body =>{
console.log(body)
callback(body, undefined)
})
.catch(err =>
callback(undefined, err))
}Let’s break that down. First, you’ll notice we have two small dependencies installed: “formurlencoded” which is used to encode objects and “node-fetch” which is used to send the request. You can install both with the npm install form-urlencoded node-fetch --save command. Why do we need to encode the object in a special way? Well the Webmention standard requires you to send the data using a “x-www-form-urlencoded” encoded string. Essentially, that means that it expects you to send data like a form on a website would, which requires a special encoding. If we want to send webmentions, we need a target URL and a source URL. Let’s say we want to send a webmention to example.org and the message we wanted to send is at nicowil.me, then nicowil.me would be our source URL and example.org our target URL. Then we’d pass both of those properties to the formurlencoded function, and it’d give us a string that looks like this "source=nicowil.me&target=example.org". The source URL contains the webmention message and the target URL is where we want to send our webmention to. The webmentionEndpoint is a URL that receives webmentions. It will usually be different from the target URL.
With that, we now know how to send a webmention! Still there’s one piece missing: what will the webmention be sending? In order for a webmention to be valid, you must include the target URL somewhere in the file located at the source URL. Typically, that means have a link to the target URL. Here’s the most minimal HTML page you could use in order to send a Webmention to this page:
<!DOCTYPE html>
<body>
<a
class="u-in-reply-to"
href="https://www.nicowil.me/posts/sending-and-receiving-webmentions-with-node">
Hello! This is my comment.
</a>
</body>
</html>As you can see, all you need to include is a link to the target URL. You’ll notice that I added a class called “u-in-reply-to” to the <a> element. That little class is an example of microformats2 which is a way to describe our web page so a computer can better understand the content. Anyways, with that knowledge in hand, let’s start building our own Webmention receiver!
Starting to build our Webmention Receiver
To kick things off I’m going to need you to install Node.js, and setup a file called index.js. You can run our program using the node index.js command. We’re going to be using Express.js to run our webserver, but the webmention code will be built separately from it so you can reuse it with whatever framework you’d like! You can install Express with npm install express --save. Since we want to keep our logic separate from our server code to prevent ourselves from being tied to Express, we’ll be creating a class called WebmentionReciever to store all our Webmention related code. Within the folder where you have your main file, be sure to create a public folder as well, I’ll explain why in just a moment. Here’s how we’ll start writing our webmention receiver:
class WebmentionReciever {
async recieve(source, target){
return {message: "Webmention Recieved!", status: 200}
}
}
const reciever = new WebmentionReciever()
const express = require('express')
const path = require('path')
var bodyParser = require('body-parser')
const app = express()
const port = 3000
app.use(bodyParser.urlencoded({ extended: false }))
app.use(express.static(path.join(__dirname, 'public')));
app.post('/webmention', async (req, res) => {
const data = await reciever.recieve(req.body.source, req.body.target)
res.status(data.status)
res.send(data.message)
})
app.listen(port, () => {
console.log("Listening on port: " + port)
})There’s nothing very special about this code at the moment. I setup the necessary class, Webmention receiver which will return a message and a status code. Since we’ll later be interacting with databases, I set up the receive function to be asynchronous. Some other key things to note is the use of the body-parser library (which you can install with the npm install body-parser --save command) so that way we can access the source and target URLs using req.body.source or req.body.target. We’re also serving the public folder, this will be where we will put our source Webmention document.
Let’s try sending a Webmention to our receiver! To do so, let’s create a file called index.html in the public folder and add the following:
<!DOCTYPE html>
<body>
<a href="http://localhost:3000/target">Testing sending a webmention.</a>
</body>
</html>You’ll notice this looks basically identical to the example I used above! I just got rid of the extra class since we won’t be worrying about microformats for this implementation. We can now reference our HTML file with the localhost:3000 URL! Try opening that page up in your browser and you should see something like this:
Let’s add the code we worked on initially which sent Webmentions to the bottom of our index.js file and have it run in the app.listen callback like this:
const fetch = require('node-fetch');
var formurlencoded = require('form-urlencoded').default;
function sendWebMention(source, target, webmentionEndpoint, callback){
fetch(webmentionEndpoint, {
body: formurlencoded({source: source, target: target}),
headers: {
"Content-Type": "application/x-www-form-urlencoded"
},
method: "POST"
})
.then(res => res.text())
.then(body =>{
console.log(body)
callback(body, undefined)
})
.catch(err =>
callback(undefined, err))
}
app.listen(port, () =>{
console.log("Listening on port: " + port)
sendWebMention("http://localhost:3000", "http://localhost:3000/target", "http://localhost:3000/webmention", function(response, error){
if(response) {
console.log(response)
} else {
console.log(error)
}
})
})Now every time our file is run a Webmention will be sent to the “/webmention” endpoint! Now you’ll notice that our target URL doesn’t actually exist, and that’s totally okay. The Webmention specification specifies that it’s up to you to reject webmentions that mention targets you don’t accept. We’ll deal with that in a little bit. If everything has worked correctly, when you run the program you should see the message “Webmention received!” appear in your console. Congrats, you’ve just sent your first Webmention!
With that we can now focus on working on our WebmentionReciever class for a little bit. You see although we did receive a webmention and could just choose to display it as is, there’s a lot of scenarios that we haven’t dealt with. Thankfully, the Webmention specification has and we can now follow it to ensure we’re in line with the other implementations and how they handle different cases. Let’s replace our current code inside the class with the following:
async recieve(source, target){
const isURLValid = this.checkURLValidity(source, target)
if(isURLValid && source != target){
return {message: "Webmention Recieved!", status: 200}
} else {
return {message: "Error", status: 400}
}
}
checkURLValidity(source, target){
return true
}The first thing we must check is to see if the URL is valid. Right now, I’m just returning true because we haven’t had a chance to dive into that yet. Next, we must ensure that the source and target URLs are different, it wouldn’t make sense for you to target your own post. If either of those conditions fail, then an error message will be sent and they will receive a failure status code of 400.
With that complete, let’s work on URL validity. A URL is valid if it starts with http or https. This is to ensure we’re only receiving web-related documents. The URL must also be included in our “valid resource” array. Remember how I mentioned that it didn’t matter if the “localhost:3000/target” didn’t exist for our demo? That’s because if a target URL fails to be mentioned in the “valid resource” array, then the webmention will fail. So as long as we include the “localhost:3000/target” URL in that array, then we’re okay! Knowing that we can now update our checkURLValidity method. We’re also going to be writing a constructor method so we can keep the “valid resource” array within the class.
constructor(){
this.validResourceHost = ["http://localhost:3000/target"]
}
checkURLValidity(source, target){
const sourceURL = new URL(source)
const targetURL = new URL(target)
if(sourceURL.protocol != "http:" && sourceURL.protocol != "https:"){
return false
}
if(targetURL.protocol != "http:" && targetURL.protocol != "https:"){
return false
}
// only other check is valid resource
return this.validResourceHost.includes(target)
}Remember that based off of our previous changes to the receive method, if false is returned from the checkURLValidity method then an error will be sent.
The final check that must occur is checking if the target URL is mentioned in the source document. The specification mentions that we should use “per-media-type” rules to determine if the target URL actually IS in the source document. For example, in a web page, the target URL should be mentioned in a “a”, or “img” tag within the “href” attribute and for a “video” tag in the “src” attribute. In a JSON document, you’d want to parse it and look for a property which contains the target URL. There is an distinction between media types because the web isn’t just home to websites, but also PDFs, documents encoded in other markup, etc. For the purposes of our implementation, we’re going to be pretending that everything is plain text and only checking if the source document includes our target URL as an exact match.
Let’s add a async verifyWebmention function that looks like this:
async verifyWebmention(source, target){
const controller = new AbortController();
// if it takes more than 5 seconds to load then cancel request
const timeout = setTimeout(
() => { controller.abort(); },
5000,
);
return fetch(source, {follow: 10, signal: controller.signal })
.then(res => res.text())
.then(
data => {
return {isIncluded:data.includes(target), err:undefined}
},
err => {
if (err.name === 'AbortError') {
return {isIncluded:false, err: err}
} else {
return {isIncluded: false, err: err}
}
},
)
.catch(err => { return {isIncluded:false, err: err}})
.finally(() => {
clearTimeout(timeout);
});
}In order for this method to work we’re going to need to install the “abort-controller” package and include it with the snippet: const AbortController = require("abort-controller") at the top of our file. Let’s break down this method, first we setup our AbortController. Why is this necessary during the verification process? We need to do our best to avoid others trying to abuse our receiver. If someone wanted to prevent others from using our receiver, they could occupy it’s time by taking too long to return the web page/file we’re looking for. In order to prevent this, if more than 5 seconds passes since the initial request for the source document, we’re going to end our request. This shields us from this attack, if someone wants to stall our receiver the most harm they can do is make us wait a few seconds extra. Another tactic to prevent others from attacking our service is to limit redirection requests. You’ll see in there’s two properties I set in the object being given to the request: follow and signal. The signal property is used to know when to stop the request, and the follow property to limit redirects. Without this protection, someone who wanted to redirect us 100 times in order to occupy time on our receiver could do so.
With those two security considerations out of the way, we can focus on what we actually care about, checking if the target URL is mentioned in the source document. If it is, then isIncluded will be set to true. In every other scenario, isIncluded will be false. We’ll use this information to change our message to the user slightly depending on the error. Now that we’ve created our verifyWebmention method, let’s add the check to our receive method!
async recieve(source, target){
const isURLValid = this.checkURLValidity(source, target)
if(isURLValid && source != target){
const isVerified = await this.verifyWebmention(source, target)
if(isVerified.isIncluded){
return {message: "Webmention Recieved!", status: 200}
} else {
return {message: "Error in verification", status: 400}
}
} else {
return {message: "Error", status: 400}
}
}You’ll notice that we don’t call our verifyWebmention method until after we’ve checked the URL validity. This is to prevent a needless request, if there’s a problem with the URLs then why even try verifying it? If the URLs pass our checks, then the verifyWebmention method will be called and return a true or false value depending on whether or not our target URL was in the source material.
With that we’ve completed our first draft of our Webmention receiver! Technically speaking, we’ve met the bare requirements for a functioning Webmention receiver. Congrats! Still our work isn’t really over. The big part of why webmentions works so well is due to microformats2. Adding this markup to our webmentions allows them to turn from humble comments, to likes, RSVPs, or almost any other sort of action you’ve come across on the web! Still, our implementation has one major flaw, which we’ll discuss in the next section.
Saving Webmentions and Asynchronous Verification
With the current method we have of verifying Webmentions, we have no defense against a user resubmitting a webmention to the same target-source pairing a million times. A bad actor who knew of this flaw in our system could take advantage of this. They could causes us to request a source URL that the bad actor doesn’t own and enable a denial-of-service attack on another person’s systems. In order to prevent this, the specification recommends that we queue each verification request and deal with each one at a random time afterwards. In order to keep a user up to date, we send them a link to a status page where they can check back for progress.
The way I’m going to do this, requires us to be introducing two more pieces of software into our receiver: Redis and MongoDB. Redis is an “in-memory” database, which we’ll be using to build a task queue with the “bull” library. You won’t need to know much about Redis and really, any task queue would work just as well. MongoDB will be used to store our Webmentions and is where we’ll update the processing information. We’ll be using the “mongoose” library to interact with Mongo, our database. In order for all of this to work you’ll need to install Redis and MongoDB and have them running.
With that out of the way, let’s get to work! Once we’ve install the packages let’s require them at the top of our file like so:
const mongoose = require('mongoose');
var Queue = require('bull');Then we’ll rewrite our constructor so we can connect to both of them as soon as our program starts:
constructor(){
mongoose.connect(process.env.MONGODB_URI,
{
useNewUrlParser: true,
useUnifiedTopology:true
}
);
this.db = mongoose.connection;
this.db.on('error', console.error.bind(console, 'connection error:'));
this.db.once('open', function() {});
this.jobsQueue = new Queue('verfiying', process.env.REDIS_URL);
this.jobsQueue.process((job, done) => {
const source = job.data.source
const target = job.data.target
this.verifyWebmention(source, target).then((isVerified) =>{
if(isVerified.isIncluded){
this.saveToDatabase(source, target)
} else {
this.saveToDatabase(source, target, true ,"Could not verify that source included target")
}
done();
}).catch((e) => {
console.log(e)
this.saveToDatabase(source, target, true, e.message)
done();
})
});
this.validResourceHost = ["http://localhost:3000/target"]
}At the start of the snippet, all we’re doing is setting up the connection to our database and our queue. You’ll notice the use of process.env.REDIS_URL and process.env.MONGODB_URI above. It’s good practice to keep sensitive data, like connection URLs which usually contain passwords and might change in different environments as environment variables instead of in our file. You can pass these strings to the node index.js command by including them at the front like so MONGODB_URI="mongodb://INSERT_URL_HERE" REDIS_URL="redis://INSERT_URL_HERE" node index.js. The most interesting section of this method is the this.jobsQueue.process part. This is the code that will run every time we add a job to the queue. This work will be done in the background. We can add a “job” by writing the line this.jobsQueue.add({source: source, target: target}). The source and target URLs will be given in the job.data object which the job uses in calling the verifyWebmention method. An important thing to note is that when the job is added to the queue it will not run immediately. It will run asynchronously. The user will need to check in at certain times, through some sort of status URL, to see when it has been processed. As of right now, we are not adding the job to the queue yet so it’s okay that the saveToDatabase method hasn’t been implemented yet.
Since we’re using Mongoose, before we define how we’re going to save things to the database, we first need to clarify WHAT it is we’re saving. We do this using a model. This is what our webmention model will look like and it should be added near the top of the file, below where Mongoose is imported.
const webmentionSchema = new mongoose.Schema({ source: String, isProcessed: Boolean, target: String, hasError: Boolean, errMsg: String ,updated: Date,date: { type: Date, default: Date.now }});
var WebmentionModel = mongoose.model('webmention', webmentionSchema)This is the bare minimum amount of information we need to process the data. What this model allows us to do is to constrain the limits of what sort of data we can insert into our database and ensures it always meets these constraints.
The following code is our saveToDatabase method which should be added to the class:
saveToDatabase(source, target, hasError=false, errMsg=null){
console.log("saving to local database...")
var query = {source: source, target: target,},
update = { updated: new Date(), isProcessed: true, hasError: hasError, errMsg: errMsg },
options = { upsert: true, new: true, setDefaultsOnInsert: true , useFindAndModify: false};
WebmentionModel.findOneAndUpdate(query, update, options, function(error, result) {
if (error){
console.log(error)
}
});
}This method will take at least two parameters, the source and target URLs. If you go back to the code we added to our constructor method, you’ll notice that we checked if the verifyWebmention method returned true or false. If the verification returns false it means it failed so we should let the user know. We do this by by saving the error message to our webmention object so the user can know something went wrong. Since there won’t always be an error, those parameters are optional. The other important thing to note about the above snippet is our use of upsert instead of insert. Essentially, this means that if a user has sent a webmention before, and now they want to update it, they totally can. This is something that needs to be allowed as a user may want to update the comment or fix an error message that resulted from their webmention.
Now to tie it all together, we need write our async version of webmention verification. Here’s what that looks like:
async verifyWebmentionAsync(source, target){
return new Promise( (resolve, reject) => {
try {
var query = {source: source, target: target,},
update = { isProcessed: false, hasError: false, errMsg: null },
options = { upsert: true, setDefaultsOnInsert: true, useFindAndModify: false};
WebmentionModel.findOneAndUpdate(query, update, options, (error, result) => {
// this will return the old object without the new update
if (error) resolve({isProcessing: false, err: error});
// if it's brand new then the result will be null, that means we can just add it directly to the queue
if(result){
if(result.isProcessed){
this.jobsQueue.add({source: source, target: target}, { delay: 5000 })
resolve({isProcessing: true, err: {message: ""}})
} else {
// if isProcessed is false then it's already being processed
resolve({isProcessing: true, err: new Error("AlreadyBeingProcessed")})
}
} else {
this.jobsQueue.add({source: source, target: target}, { delay: 5000 })
resolve({isProcessing: true, err: {message: ""}})
}
});
} catch (e){
reject({isProcessing: false, err:e})
}
})
} Since this method will run asynchronously we’ll return a Promise explicitly. Then we try to look for the webmention and using upsert, either create or update it. We update the isProcessed property which we’ll use to determine if a webmention is currently in the job queue and in processing. We’ll also clear away previous errors by resetting the hasError and errMsg properties. If there’s any sort of error that occurs in this, we’ll catch it and tell the user. Next, we use the result data that is returned from our database query. In the else clause we deal with the scenario where the result is null. That happens when the webmention that’s been sent is brand new, so we can directly add it to the queue.
If the webmention does exist, then it could be that the user wants to update it or they may have realized that an error occurred in the verification step so they’re sending it again. Regardless of why they want to change their webmention, we check if the webmention has been processed already. If it’s in processing (meaning isProcessed will be false), then we return an error to the user. This is because we don’t want to be processing the same webmention more than once at a time. The user will need to wait until it’s been processed before sending it to us again. Otherwise, it’s ready to be added to the queue. You’ll notice that when we add the job to the queue we add a delay of 5000 which means we want the job to only run after 5000 milliseconds (5 seconds). If you wanted to use this code in your own implementation, you’d probably want to make that time fluctuate a bit randomly and think a bit more closely about what you want your delay to be.
Now it’s great that our webmention is going to be processed asynchronously, but how will the user know when it’s done being processed? We can do that by adding a status method which will just return the webmention object that the user is looking for. Here’s what our status code should look like:
status(source, target){
return new Promise( (resolve, reject) => {
WebmentionModel.findOne({ source: source, target: target },function(err, webmention) {
if (err){
console.log(err)
reject(err)
};
resolve(webmention)
});
})
}All we do is look for a webmention with a specific source and target URL and return it. Since we upsert all of our webmentions, it means there will only ever be one webmention with a particular source-target pairing.
Let’s add the following snippet to the bottom of our file so we can have a URL which will call our status code.
app.get('/status', function(req, res){
const source = req.query.source
const target = req.query.target
reciever.status(source, target).then((msg) => {
if(msg){
res.json( msg);
} else {
res.json({err: "That source-target combination does not exist. Please make sure you have entered them correctly or correctly sent a webmention."})
}
}).catch((e) => {
console.log(e)
res.status(400)
res.json({err: "An error occured. Please try again later."})
})
})We’re getting so close to the finish line! With that being done, we need to tie all together so we will no longer use the synchronous version of our verifiyWebmention method and replace it with the async version. Here’s what our new receive method will look like:
async recieve(source, target){
const isURLValid = this.checkURLValidity(source, target)
const statusURLBase = "http://localhost:3000/status"
if(isURLValid && source != target){
try {
const isMentionedCheck = await this.verifyWebmentionAsync(source, target)
if(isMentionedCheck.isProcessing && isMentionedCheck.err.message == ""){
const statusURL = `${statusURLBase}?source=${encodeURIComponent(source)}&target=${encodeURIComponent(target)}`
return {message: "You can check the progress at " + statusURL, locationHeader: statusURL, status: 201}
} else {
const err = isMentionedCheck.err
return {message: err.message, locationHeader: null, status: 400}
}
} catch(e){
return {message: e.message,
locationHeader: null, status: 400}
}
} else {
return {message: "Error",
locationHeader: null, status: 400}
}
}This is pretty similar to our old receive method. Instead of just running the verifyWebmention method, we now use our async version. We’ve modified our message and added the necessary code to catch any errors that may arise in the async verification process. You’ll notice that we are now telling the user the URL they can use to check back for updates, and we changed the status code from 200 to 201. The source and target URLs are both put through the encodeURIComponent function to make sure they’re safe as URL query parameters in order to create the status link.
The Webmention specification outlines that if we want to tell the user that there’s a URL they can use to check back for updates, we should set the Location header to contain the status URL. Let’s do that now by modifying our webmention receiver endpoint to look as follows:
app.post('/webmention', async (req, res) => {
reciever.recieve(req.body.source, req.body.target).then((data) => {
res.status(data.status)
if(data.locationHeader){
res.set('Location', data.locationHeader)
}
res.send(data.message)
}).catch((e) => console.log(e))
})Congrats! You’ve successfully implemented your own Webmention receiver! It’s taken us a while, but we did it! So although this is cause to celebrate, we should note that we can add a couple of things in order make it more in line with the specification such as adding deletion of webmentions and properly checking the status types when verifying our webmentions. Other than those loose ends, we did it! Now that you’ve successfully completed this challenge, I highly recommend you take a look and read the specification! It’s really interesting and goes into much finer detail that we did here. I hope that this was an enjoyable project for you and you learned more about webmentions! If you have any remaining questions, feel free to reach out. Have an amazing day and until next time!