How to Index a Free Tier Ngrok Website With Coveo

How to Setup an Ngrok Site as a Coveo Source

I was given a task to create some test data in Coveo that we could use to build a prototype search interface. We wanted to do this by building a demo site that would feed some dummy data into a Coveo source. The site ran locally on my machine, but most Coveo sources want a public domain to read from. Someone mentioned ngrok as a solution that might solve my problem.

The key feature we used from ngrok is its ability to create a secure tunnel to open access to remote systems, bypassing the need to touch any network or router settings. This allowed us to turn our demo website public so that Coveo could read it.

Maybe you’re in a similar scenario where you just want to make a quick prototype or want to learn Coveo using your own projects. This guide will help you get a locally running project working with Coveo and ngrok.

Step 1: Get a Site Running Locally

The example website we’ll use is a very basic, plain HTML website about rocks. Each subpage is about a particular type of rock and contains the following information in its metadata: the rock’s name, a description of the rock, and which category it falls under (Igneous, metamorphic or Sedimentary). I’m running it locally using the Live Server extension for VS Code. Make a note of the port number (in this case, it’s 5500).

Step 2: Download Ngrok And Make Site Public

Download ngrok for your system. For this case, I downloaded the Windows 64-bit zip folder, extracted it, and double-clicked ngrok.exe. Then I ran ngrok -v to make sure it was working. You don’t really need to create an ngrok account for this demo, but it is advisable to do so later because then you can get an auth token that can keep your site public for longer.

Now that ngrok is ready to go and we have our site running locally, let's make our demo site public.

In your ngrok command window, enter ngrok http <your port number>. Our rock example is running on port 5500, so I’ll run ngrok http 5500.

The command window will update, and you should see something like this:

The part underlined in red is your public site link. Go to that URL, and you’ll see a warning page.

This warning page will block access from Coveo when it tries to index the site. As it says in the message, there are a couple of ways around it: upgrade to a paid ngrok account, or add a User-Agent request header. The option of sending an ngrok-skip-browser-warning request header won't work for more modern sites that use JavaScript (you can’t use it if you have the Run JavaScript setting in your Coveo source settings on).

But the UserAgent header on Coveo will work with plain HTML and modern JS applications, so we’ll use that in the next step.

Step 3: Create a Coveo Source from the Ngrok Site

In Coveo, create a web source and point to your ngrok website.

Click through, and you’ll come to more options. Depending on your site, you may have to set more options. For this demo site, I need to set the following under Configuration > Advanced Settings.

Click Save, then click Sources under the Content menu in Coveo. Find your source, single-click on it, and from the top menu, select More > Edit JSON.

In the JSON, under parameters, add the following.

"UserAgentString": {
    "sensitive": false,
    "value": "allow"
},

Click Save and rebuild source. After it builds, check out your source in the Content Browser, where you should now see your pages.

These Coveo items can now be used to build a prototype search interface. From here, you may want to explore how to use your data to create an Atomic search interface or learn about field mappings in Coveo.

---